Live Reindexing with Elasticsearch

In order to upgrade Elasticsearch in-place from v2 to v5, one of the road blocks we faced was that the built-in upgrade feature did not support indexes created with v1 of Elasticsearch.

We had several of those left over from a previous upgrade from v1 to v2. So prior to upgrading we began a procedure of reindexing to replace the v1 indexes with new ones created inside v2. This procedure can also be used to change mappings in ways that are not supported by in-place "PUT mapping".

Pre-flight: Backup!

Before starting, its a good time to check that your Elasticsearch backups are working. At Chargify, we use official AWS S3 plugin for Elasticsearch snapshotting, and we've automated a daily test restore to a secondary one-node cluster.

Overview

Create an alias

First up, if you haven't already, you need be reading from Elasticsearch via an alias instead of directly accessing an index by its name.

Begin tracking source index name

Since you'll have multiple indexes containing the data, it starts to matter where the data came from. When loading a document, Elasticsearch reports which index it came from as metadata. Begin tracking this, so you can save a changed document back to the index it came from, or if you are deleting the document, you delete from the correct index name.

If you only ever insert new documents, this step can be skipped.

This would also apply to bulk operations, if previously you did bulk updates or deletes (without supplying an index name per bulk op), then this needs to change.

Fall-back single document ops to the old index

The alias feature allows searches to come from more than one index at a time. But aliases are not used for single document operations, ie. plain GET by ID.

Two possible solutions:

• Via code, retry GET on 404, attempting on each of multiple index names (new and old)
• Instead of using the document GET API, use a search (with size: 1) instead.

Using a search is easier, but remember that new documents appear instantly via GET, but only appear in a search after the standard indexing delay (unless you force a refresh).

Writing to a new index

The next step is to create a new index, and add it to the alias. New documents should be written to the new index, and reads of the alias will see data in both the old and new index.

This new index will hold documents created from that point onwards. This removes indexing load from the old index, allowing you to duplicate it.

Duplicate the old index

As a background job, we now need to duplicate the old index into a new one. Elasticsearch has a built in function called Reindex to do this, or you can write your own which just retrieves and stores documents in bulk.

If you writing your own, pay attention to the document version numbers, in case its important to you to keep the original version numbers in the new index.

You may also like to write some code to verify the duplication has worked. At its simplest you could just compare document counts between old and new, as a sanity check.

Swap to the new index via atomic alias change

Once reindexing is complete and verified, it is time to stop using the old index. Elasticsearch can make multiple changes to an alias atomically, so you can remove the old and add the new without worrying about a race condition causing downtime or duplicate search results.

Clean-up

Congratulations! You are now finished reindexing. The final step is to remove the old index, as well as removing any fall-back application code that references it.