UPGRADING ELASTICSEARCH

WITH ZERO DOWNTIME

Best Practices and Lessons Learned from SignalFx Engineering

@signalfx | www.signalfx.com
 Table of Content
Executive Summary ........................................................................................................................... 3

Why Elasticsearch .............................................................................................................................. 3

Why Upgrade Elasticsearch ............................................................................................................... 3

Challenges with Upgrading ............................................................................................................... 4

Non-Backward Compatibility .................................................................................................... 4

No Support for Rolling Upgrades .............................................................................................. 5

Zero Downtime Upgrade in Practice ................................................................................................ 6

Client Abstraction and Switching Mechanism ........................................................................... 6

Key Components of the Indexing Pipeline ................................................................................. 8

A Step-by-Step Process for Zero Downtime Upgrade ................................................................ 9

The Indexing State Structure .................................................................................................... 9

Phase 1: Initialization Phase ................................................................................................. 10

Phase 2: Bulk Import Phase .................................................................................................. 11

Phase 3: Double Publishing and Cleanup Phase ..................................................................... 12

Phase 4: Closing Phase ......................................................................................................... 14

Case Study: Performance Comparisons between Elasticsearch 1.7.5 and 2.3.3 ......................... 16

The SignalFx Environment ...................................................................................................... 16

Indexing Performance ............................................................................................................ 17

Query Performance ................................................................................................................ 18

Filter Cache ............................................................................................................................. 19

CPU Utilization ........................................................................................................................ 20

Conclusions ............................................................................................................................ 20

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 2

 Executive Summary
Elasticsearch has progressed rapidly from version 1.0.0, released in February 2014, to version
2.0.0, released in October 2015 and version 5.0 released in December 2016. During the two-and-
a-half-year period since 1.0.0, adoption has skyrocketed, and both vendors and the community
have committed bug-fixes, interoperability enhancements, and rich feature upgrades. This
has made Elasticsearch one of the most popular distributed search engines for unstructured
documents, as well as a great log analysis tool as part of the ELK Stack. However, despite
major improvements with each release, the majority of organizations continue to use the same
version as when they initially adopted Elasticsearch—in most cases, version 1.x.

Since the launch of SignalFx in early 2015, Elasticsearch has been core to meeting the search
requirements of time-series metadata that drive catalog suggestions, alerts timeline and most
importantly applying analytics in real-time to the metrics and enabling intelligent alerting that
cuts down the noise. These are the best practices and lessons learned from the entire process of
upgrading Elasticsearch with zero downtime.

Why Elasticsearch
Elasticsearch is an open-source distributed search engine based on Apache Lucene. It is highly
distributed and designed for easy implementation, fast queries against large data volumes,
multi-tenant availability, and horizontal scale. Elasticsearch is used primarily as a NoSQL
storage, indexing, and search utility for unstructured documents and can also serve as a log
analysis tool as part of the Elastic Stack.

Elasticsearch is highly scalable, providing a great API, and very easy to work with for all
engineers. Ease of setup also makes it very accessible to developers without much operational
experience.

Why Upgrade Elasticsearch

Updates to Elasticsearch versions have focused on resiliency, reliability, simplification, and
performance, as well as new features and enhancements. For the SignalFx engineering team,
the most relevant and interesting features of the newer Elasticsearch version included:

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 3

 • Resiliency Improvements to ensure no data loss with Elasticsearch.
• Dynamic Merge Scheduling with an auto-regulating feedback mechanism. This eliminates
past worries around manually adjusting merge throttling settings and allows Elasticsearch
to provide more stable search performance even when the cluster is under heavy
indexing load.
• Synched Flushes to enable instantaneous recovery of existing replica shards. In previous
versions, node failures or a reboot can trigger a shard allocation storm, and entire
shards are copied over to the network despite already having a large portion of the data.
Recovery times have been reported at over a day to restart a single node.
• More Lenient Reallocation to avoid unnecessary reshuffling during node restarts. Based on
experience, this works well if the indices are primarily read-only (for example, for time-
based indices).
• Memory Pressure Management and Better Handling of Expensive Queries which was limited
in the 1.x versions of Elasticsearch.

Challenges with Upgrading

Despite the vast improvements across Elasticsearch with recent releases, the updates come
with shortcomings in terms of the ease of upgrading legacy implementations and breaking
changes. These challenges are growing pains in the context of the performance enhancements
and bug fixes that result from an upgrade, but they still require attention during any transition.

Non-Backward Compatibility

New releases of Elasticsearch has introduced features that may affect compatibility between
versions and requires adoption of new approaches to existing workflows. A few of the more
difficult incompatible mapping changes include:

• index_analyzer setting have been removed to favor analyzer.

• _analyzer metadata field is also removed.
• Field names with dots are not supported anymore.
• String fields are now replaced by Text for analyzed and Keyword for non analyzed strings.
• _timestamp and _ttl fields are no longer supported.

A second drawback to upgrading Elasticsearch is the incompatibility between the Java client of
version 1.x or lower and the server of version 2.x or higher, and vice versa.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 4

 Examples of the key pain points to be handled by the client side include:

• FilterBuilders are replaced with QueryBuilders.

• Filtered query is removed along with or, and, and missing queries.
• TransportClient has been moved to its own module.
• Count API is no longer supported.
• Suggest API is removed.
• Few of the exceptions have been removed (such as DocumentAlreadyExists and
IndexMissingException).
• Signature changes for few of the request builders.
• Write consistency levels have been removed from the write requests.
• Plugins now need to have a plugin-descriptor.properties file. Failing to have the
file will cause the node not to start.

No Support for Rolling Upgrades

Rolling upgrades are not supported while upgrading an Elasticsearch cluster across major
versions. Hence, it requires a full cluster restart upgrade. There are several things to consider if
opting for a full cluster restart upgrade:

• A full cluster restart upgrade inherently requires downtime of the entire cluster. Shutting
down the cluster for an upgrade means that indexing or querying for that duration is
impossible. This should be planned in advance and coordinated with the services or
applications that rely on the cluster being available.
• Once an Elasticsearch server version has been updated from one major version to another,
data cannot be rolled back to the previous format. It is not possible to go back once an
upgrade has occurred.
• There cannot be any mapping conflict inherited from the current index.
• If data striping is used to store data from an index on multiple paths, make sure the path
has enough disk space. Unlike version 1.x, data from a single shard can only be stored on
one path. This is worrisome if document distribution is not uniform across shards or if
there is a single shard per node.
• All plugins now require a descriptor file, so check if current plugins are compatible with
the upgraded version. Install the latest supported plugins on each node.

If possible, it is recommended to set up a new cluster and to re-index data to reflect the new
mapping rules. With a new cluster, there is flexibility to redesign document modelling strategies
to create a separate index in case of conflicts in field names or if the types from the index are
deleted.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 5

 Zero Downtime Upgrade in Practice
The approach for performing an Elasticsearch upgrade with no downtime has the flexibility to
continue serving incoming indexing and querying requests while allowing roll backs, in the case
of failures.

Client Abstraction and Switching Mechanism

At SignalFx, the Elasticsearch client is hidden behind a library that is used across all services. A
series of key steps were taken to make the library capable of communicating with two different
clusters and completely abstract that from its users. This is a required step in any migration that
requires a fallback to the old version in case of failures.

signalfx-search-lib

es-1.7.3-jar

Abstracting the Elasticsearch Client through a Wrapper

Step 1: Bundling the Two Client Versions

It is required to have two versions of the client libraries due to the backward incompatibility
and the need to talk to both versions of Elasticsearch at the same time.

To deal with the namespace conflict between the two libraries, the Apache Maven Shade plugin
is used to rename one of the Elasticsearch clients namespace. For example, renaming “org.
elasticsearch” to “org.elasticsearch2”. Now both clients can be used within the same
JVM.

signalfx-search-lib

es-1.7.3-jar shaded-es-2.3.3-jar

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 6

 Step 2: Building a Client Agnostic Abstraction

Next, the two Elasticsearch client implementations need to be abstracted behind a common
interface. This interface not only abstracts away the implementation difference between two
clients but also exposes the minimum set of APIs required by the users to talk to Elasticsearch.
This requires boilerplate POJOs, Builders, Exceptions, etc. that are needed for a fully functional
API.

signalfx-search-lib

es-wrapper-lib

es-1.7.3-jar shaded-es-2.3.3-jar

Implement the interfaces for both versions of Elasticsearch. The implementation is identical
except in the areas mentioned in the “Non-Backward Compatibility” section.

The Overall Process

The easiest way to upgrade an Elasticsearch cluster is to do it in place. Nodes are shut down,
their ES binaries upgraded and then restarted with the new version. This procedure may
take hours and requires a service downtime. There are a lot of cases where downtime is
unacceptable, and implementing the upgrade without a downtime is the desired approach.

Since taking down the current cluster is not possible for zero downtime requirements, it is
necessary to stand up another cluster with the required Elasticsearch version and copy the data
between the two clusters. This has the following implications:

• More storage capacity is required to handle the storage requirements of the new cluster.
• While copying the data from the current cluster to the new one, data will change. New
documents can be added, existing documents may be updated or deleted. At the end of
the process, it’s crucial that no writes get lost.
• Copying large amounts of data takes time and computing resources. The longer this takes,
the more likely failures will occur. The process must be resilient to such failures.
• If a zero-downtime upgrade is a requirement, it’s likely that the Elasticsearch cluster is
a critical part of the overall environment. It’s important that such an upgrade can be
canceled at any moment. The load on any production cluster should also not impact its
proper functioning.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 7

 At a high level, a zero-downtime Elasticsearch upgrade process includes:

1. Provisioning a new cluster with the desired version of Elasticsearch.

2. Copying a snapshot of the data from the existing cluster to the new one. This can be done
through a direct copy (such as using the scroll API) or by restoring a snapshot from an
existing backup.
3. Starting “double-writing”. Any document indexed on the current index will also be present
on the new one.
4. Reconciling any changes that happened to the current index during steps #2 and #3 to
ensure that no data is lost.
5. Pointing clients to the new index.

Key Components of the Indexing Pipeline

For SignalFx, the key components of the Elasticsearch indexing pipeline include:

• Indexer: responsible for writing or indexing documents into Elasticsearch. It can be a

separate service or a library that other components use internally. There can also be
multiple indexers for high availability or for scale-up reasons.
• Reader: responsible for issuing queries against Elasticsearch. This can be either a library
or a service that acts as an abstraction layer on top of Elasticsearch queries. This layer can
completely hide the Elasticsearch semantics behind a higher level query engine. It can
also be a thin wrapper around the Elasticsearch APIs. There can also be many readers,
which is more likely to be the case should Elasticsearch hold data that needs to be queried
from multiple services.
• “Indexing State”: a global datastructure that must be accessible by both the readers and
the indexers. It allows them to know where the cluster is and which index to query. This
can be as simple as a hardcoded host and index names, or a more sophisticated dynamic
configuration that can map various queries or documents to different indexes or even ES
clusters. This state must be visible to all parties at all times.
• Migration Manager: a component that will coordinate the migration process; creating the
index, copying the data and reconciling any changes that happened during the upgrade.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 8

 A Step-by-Step Process for Zero Downtime Upgrade

The SignalFx team has implemented and successfully executed a zero downtime upgrade
multiple times. While this process is specific to SignalFx, these lessons can be generalized and
applied to any similar indexing pipeline.

The Indexing State Structure

SignalFx’s indexing state contains the following information:

• Current Write Index Name: The name of the index currently being written to.
• Current Write Cluster Name: A logical name to determine what cluster to write to. At
SignalFx, there is a home-grown service discovery mechanism to get an Elasticsearch
client to any cluster by providing the cluster’s logical name. The name can be descriptive,
such as “elasticsearch-prod-cluster” or “elasticsearch-5.0-prod-cluster”.
The combination of the index name and cluster name uniquely and unambiguously
identifies any index within the infrastructure.
• Current Query Index Name: The name of the indexing (or alias) to query from. This is
typically the same as the current query cluster name.
• Current Query Cluster Name: The logical name of the cluster used for queries.
• Extra Index Name: The name of an extra index that will receive the same writes as received
by the main index. This index is not used for reads, and in normal situations, this field is
unset.
• Extra Cluster Name: Similar to the main cluster name and allows the ability to properly
locate the extra index. This is also not set in normal situations.
• An Index Generation: A monotonically increasing number. Any document written will have
a special field “_generation” that contains the value of the current index generation.
All documents in the index must have this field. Different documents in the index do not
necessarily belong to the same generation number. In addition, this number does not
change in normal situations—it is only incremented only as needed.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 9

 Reader Indexer

current: es-prod/index
generation: 42 Index
extra: <null>

For SignalFx, the indexing state is stored as a JSON blob in Zookeeper. Readers and Indexers
can access this data structure very efficiently (cached in-memory) and it is used to determine
how a query or an indexing operation should be executed. Indices are written as “<cluster-
name>/<index-name>”. For example, the SignalFx main index is called “es-prod/metrics”.

It is important to ensure that changes to the indexing state are properly seen by all indexers.
This ensures that any guarantees the migration tool is relying on are satisfied at all times.
SignalFx uses Zookeeper for this coordination. When the indexer state changes, all the indexer
nodes will write the state version to another Zookeeper node. The migration tool will not move
forward until all the indexers have acknowledged that they have seen and applied the change
(the number of indexers is known ahead of time).

Phase 1: Initialization Phase

This phase involves standing up a new cluster and creating the target index. When upgrading to
a major version of Elasticsearch, there are no guarantees on the backward compatibility of the
index mapping configuration. For example, Elasticsearch 2.x mappings are not compatible with
1.x. While changes may be syntaxical changes, they are sometimes more subtle (for example,
doc values becoming the default or a deprecated feature is completely removed). This must
all be tested on a test/staging environment to fully understand the impact of any differences
between the mapping versions.

Indexer

current: es-prod/index
generation: 42 Index Index
extra: <null>

current cluster: ES 1.7 new cluster: ES 2.3

Because the new index will not be queried yet, it is best to disable refreshes on it by setting
the refresh_interval to -1. It is also not required to have any replicas for the new index.
However, it is recommended to have at least one replica so that a host failure can be dealt with
without requiring a restart of the migration from scratch.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 10

 It is typical to spin up new nodes to host the new cluster. This can be costly if the cluster size is
large (for example, hundred of nodes). Capacity planning should be done carefully to ensure
production SLAs are not impacted.

Phase 2: Bulk Import Phase

During this phase, documents are copied from the source index to the target index using the
migration tool.

To handle concurrent changes to the source index while the copy is taking place, the migration
tool increments the generation number before beginning of the copy process. In the diagram
below, the generation number is incremented from 42 to 43.

As documents are updated or added, their generation will have the 43 value. The copy process
will only need to copy any documents that have a generation less than or equal to 42, and this
is guaranteed to be a bounded set. It will either stay the same or increase in size but will never
increase. This ensures that the copy operation will finish, regardless of how many documents
are updated/added during the migration process.

Indexer

current: es-prod/index
generation: 43 Index Index
extra: <null> _generation <= 42

The most efficient way to do the copy operation is to use Elasticsearch scroll API. The scroll API
has a few limitations:

• Scrolling can disallow merged segments from being claimed back by the filesystem since
it keeps those segment’s file descriptors open. This can be problematic if the segments are
large and there is insufficient storage capacity overhead.
• If scrolling fails and retries do not happen fast enough, the scroll will expire. This makes
the recovery painful since it requires the scroll to be restarted from the beginning. On large
indices, restarting the procedure can be costly (including network cost, maintaining two
clusters at the same time, etc.).
• There is no easy way to “pause” the scroll and resume it later.

To avoid these pitfalls, the documents are partitioned into multiple buckets. A document
is assigned a _bucket_id when it is created. The _bucket_id is based on a hash of the
document id modulo the number of buckets. The _bucket_id is also written as a field on the
document (similar to the _generation field) in order to be used by the migration tool for its

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 11

 queries.

This partitions the document space into a uniformly distributed set of buckets, which is added
as a parameter to scrolling—in effect, preventing scrolling over the entire index at once.
SignalFx partitions its id space into 216 buckets. This allows the ability to migrate indices of
billions of documents in smaller chunks of about one million documents each.

The bucketing mechanism also allows the bulk migration to be recoverable in case of failures.
Since bulk copies of large indicides can take days, there can occasionally be problems that call
for pause, recovery, or rollback. Restoring from the last successful bucket is a lot better than
having to redo the whole thing.

At the end of this phase the indices will look like this:

Indexer
Index Index

43
43

current: es-prod/index
generation: 43 43

extra: <null> 43

43

Notice the new document as well as the updated documents with _generation 43. These are
the documents that changed during the migration process and that will have to be reconciled in
later phases.

Phase 3: Double Publishing and Cleanup Phase

At this stage, all the documents with generation number 42 or less are already migrated to the
new Elasticsearch cluster. Documents with a generation number higher than 42 need to
be reconciled.

To start the reconciliation process, it is necessary to handle two issues:

1. Copy any document with _generation 43 to the target index.

2. Handle documents that changed while completing the previous step (for example,
documents at revisions 42 or less and 43 can still be changed and those changes must not
be lost).

To deal with the second issue, use the same technique as in the previous step: the generation
number. However, ensure that documents written going forward are written to both indices.
Otherwise, this process will have to be done over and over again without ever catching up. This
is where the extra index name and cluster name come into play.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 12

 Double Publishing

The first step in this phase is to instruct the indexers to start writing to both indices. To do
that, the migration tool will update the indexing state and set the extra index to es2.3-prod/
index. In addition, the generation number is bumped up from 43 to 44. Going forward, this
means any update or addition will be written to both indices with a generation number of 44.
More importantly, the set of documents at generation 43 is now a bounded set—it either stays
the same size or shrinks (as documents are updated and their generation changes to 44).

Indexer
Index Index

43
43
current: es-prod/index
generation: 44 43
extra: es2.3-prod/index 43 43

43

43

Cleanup

Now, it is only needed to reconcile all the documents at generation 43. To reconcile those
documents, it is necessary to re-write them at generation 44. Since reading and re-indexing the
document is not atomic, an update can get in between and be lost if the document is written
after the update.

To handle that case, Elasticsearch optimistic concurrency control can be used. If there is a
version conflict, ignore it. A version conflict indicates that the document has been concurrently
written by an indexer, which means it has been written with 44 as the generation number since
double publishing has been enabled in the previous step.

At the end of the cleanup phase, there should be two identically looking indices on both clusters.

Indexer
Index Index

44 44
44 44
44 44
current: es-prod/index
generation: 44 44 44
extra: es2.3-prod/index 44 44 44 44

44 44 44 44

44 43

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 13

 Phase 4: Closing Phase

Now that the indices are exactly the same (modulo some delay in indexing due to its
asynchronous nature), it is possible to start changing the query path while keeping double
publishing enabled.

Before doing that, the number of replicas is gradually increased to the same level as the original
index. Based on the SignalFx experience, index migrations are typically run either with no
replica or with a single replica. The replica factor is only increased before the index is open for
production traffic queries.

In general, it is preferable to switch only a portion of traffic or even better a set of specific users
to the new index/cluster to make sure that everything looks good before switching everybody.
At SignalFx, there is the capability to switch a specific organization (i.e., a customer) to the
new index while other organizations are still on the old index. This allows switching the testing
organization in the production environment to the new index to sanity check that things are
properly working before switching everybody else.

SignalFx has also implemented a cluster warm-up procedure where a portion of the query
traffic is sent to both the old and new clusters. The results from the new cluster are ignored.
The goal is to warm up the new cluster so that caches are ready when the final switch occurs. In
practice, it has not been necessary to use this feature.

Verification

The readers are switched to the new index/cluster. During this step, it is possible to switch
back and forth the read pointer to any index in any cluster. While this has not been needed in
practice, it provides the option should an issue be discovered with the new index.

Reader Indexer

Index Index

44 44
44 44
44 44
current: es-prod/index
generation: 44 44 44
extra: es2.3-prod/index 44 44 44 44

44 44 44 44

44 43

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 14

 Finalizing

To finalize the migration, the last remaining steps include:

• Stopping double publishing. This is done by setting the current index to the extra one
and unsetting the extra index. In practice, it is typical to bump up the generation number
one more time in order to handle the case where a rollback occurs to the previous cluster
after stopping double publishing. By having all documents after the switch have a higher
generation number, only those would need to be migrated back to the previous cluster.
• Closing the source Elasticsearch 1.x index to free up resources. Best practice is to retain the
index for a week before completely removing it and decommissioning the old cluster.

This is how the system will look like at the end of this phase:

Reader Indexer
Index

44
44
44
45 45
current: es2.3-prod/index 44
generation: 45 44 44
extra: <null>
44 44 45

44 45

What about Deletes?

The current process does not properly handle deletes. If a document is deleted during the
first phase, for example, it will be orphan in the target index (namely, it gets resurrected). For
SignalFx, most deletions in the system is triggered by the metric time series expiration policies.
During the upgrade, the expiration process is disabled. In the future, it is possible to look at
either a deletion log to re-apply post upgrade or a deletion marker in Elasticsearch and a
process at the end of the migration to cleanup objects with the marker.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 15

 Case Study: Performance Comparisons
between Elasticsearch 1.7.5 and 2.3.3
SignalFx is known for monitoring modern infrastructure, consuming metrics from things like
AWS or Docker, applying analytics in real time to that data, and enabling alerting that cuts
down the noise. Core to how SignalFx does that is search. It is insufficient to simply process
and store/retrieve data faster than anything out there, if it takes a long time for users to find
the data they care about. To match the speed of the SignalFlow analytics engine that sits at the
core of SignalFx, Elasticsearch has been used for all searching needs from day one.

As the backbone to serve the needs of high-performance, production applications, the SignalFx
engineering team consistently strives to optimize their overall infrastructure. With the release
of Elasticsearch 2.0 in late 2015, the team invested the time and resources to update their
production cluster from Elasticsearch 1.7.5 in order to continually optimize the search needs
of a growing customer base. These are the performance results observed after upgrading to
Elasticsearch 2.3.3 in production.

The SignalFx Environment

The SignalFx Elasticsearch production cluster has 54 data nodes with more than 2 billion
documents and a data size of 20TB (including replica). The cluster runs on i2.2x large AWS EC2
instances with 16GB heap solely dedicated to Elasticsearch. The 54 nodes are spread across
three availability zones. Three copies of the data are maintained, one in each availability
zone (ie, one primary and two replica shards). There is also a dedicated client node tier with 6
m3.xlarge (10GB of heap) and a dedicated master node tier with 3 m3.large (2GB heap)

Before and after the upgrade, the cluster had the same number of indices, the mappings were
not changed and the document count did not change significantly.

As part of the Elasticsearch upgrade, the client and master nodes for the new cluster were
hosted on separate instances. On the other hand, the data nodes were co-located with the data
nodes of the existing cluster. Using a Dockerized version of Elasticsearch made it easy to run
both versions on the same instance. The team made the determination that there was enough
capacity (storage, IO, CPU, Memory) to handle both data nodes on the same instance. This was
possible because both clusters were not queried at the same time, and queries were the biggest
portion of CPU usage. Furthermore, there was the ability to control the indexing load on the
cluster through throttling of the indexing pipeline, which may cause a slight delay in indexing
unnoticeable in practice.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 16

 Indexing Performance

SignalFx generates more than 100 million index requests on a daily basis. The documents in
the Elasticsearch cluster are mutable, meaning indexing requests also include updates to the
existing documents in addition to new documents.

The chart below displays the rate of indexing requests received by both clusters. Once the
double publishing phase started, the 2.3.3 index started getting indexing requests similarly to
the 1.7 index. However, since there are only primaries on the target index, the indexing rate is
lower (one third). Once the replica count was increased to 2, both Elasticsearch 1.7.5 and 2.3.3
start indexing at the same rate. At the end of the migration, double publishing is stopped, which
is seen in the graph below as a complete stop in indexing requests on Elasticsearch 1.7.5.

The chart below displays the mean indexing time for both clusters during the double publishing
phase after replicas were added to the target Elasticsearch 2.3.3 cluster. At this time both
clusters receive the same indexing requests. The chart suggests that indexing takes longer in
Elasticsearch 2.3.3 (orange) as compared to Elasticsearch 1.7.5 (green).

A possible explanation for the higher indexing latency in Elasticsearch 2.3.3 is the new
synchronous fsync behavior. In Elasticsearch 2.x, translog is now fsync’ed after every write
request in addition to the previous frequency of every 5 seconds. The fsync takes place on both
primary and replica shards. The client does not receive a 200 response code until the request
is fsync’ed, hence the increase in indexing latency. The behavior can be changed to make the
fsync asynchronous.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 17

 Query Performance

The catalog search in the SignalFx app is powered by Elasticsearch and showed blazing fast
speed immediately after upgrading to Elasticsearch 2.3.3. These searches include complex
queries with aggregations in addition to the suggestions provided as part of the search.

In the chart below it can be seen that the query response time for Elasticsearch 1.7.5 (in green)
cluster varies between 300ms to 800ms, occasionally peaking as high as 1000ms, while the
query response time for the new 2.3.3 cluster (in orange) consistently remains in the lower
range of 200ms to 400ms. Also the response time is less jittery after upgrade.

To compare the mean query latency for the two versions running at different points in time, the
query response time of the Elasticsearch 1.7.5 cluster is time-shifted by six weeks. The chart
displays the comparison of mean search time over the course of one day for Elasticsearch 1.7.5
(in green) and Elasticsearch 2.3.3 (in orange) clusters.

The mean query time for Elasticsearch 2.3.3 cluster is observed to be 4ms with a P99 of 5ms,
compared to the mean query time of 7ms and a P99 of 9ms for 1.7.5. The improvement in mean
search time could be an outcome of various improvements made in Elasticsearch 2.x query
execution engine.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 18

 Filter Cache

There is a drastic reduction in the filter cache size, which is a major highlight of the transition
to Elasticsearch 2.3.3. In Elasticsearch 1.7.5 (in green) where filter cache was a configurable
setting, the limit is set to 20%, which is 3.2GB for SignalFx.

SignalFx heavily uses filters in its queries and for performance reasons, the filter cache was
enabled and frequently used. After switching to Elasticsearch 2.3.3 (in blue), the filter cache
size dropped to a miniscule 18kB, representing a whopping 99.999% reduction from the
original 3.2GB.

In the chart below, it is clear that the SignalFx application experienced a high thrashing rate of
the cached results. As the customer base grew and as more and more queries are processed, it
is expected to observe a slight increase in slowness.

Despite dedicating time and effort in the past to optimize filter cache usage across our services,
the filter cache thrashing rate did not improve. After switching to Elasticsearch 2.3.3, the cache
eviction rate reduced from a mean of 55 per second with Elasticsearch 1.7.5 (in green) to a mean
of 9 per second. This represents a massive improvement in query execution efficiency and has
been a major driver of the unprecedented performance observed in the SignalFx production
cluster.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 19

 CPU Utilization

The chart below shows CPU consumption for the same time periods as in the indexing load
comparison chart in the earlier section. CPU consumption with Elasticsearch 1.7.5 (green)
is approximately 20% with many more sporadic spikes, impacting both indexing and query
latency. On the other hand, Elasticsearch 2.3.3 CPU utilization (orange) remains below 10% for
the majority of the time. It only gradually rises towards 20% with an increase in the indexing
load and does not show any of the troublesome spikiness observed with the 1.7.5 cluster.

Conclusions

For the SignalFx team, the upgrade to Elasticsearch 2.3.3 was worth the engineering efforts to
deal with the version incompatibilities and run a migration on an index of tremendous size.
There are significant improvements in all performance metrics after upgrading to Elasticsearch
2.3.3­—search became much faster after the upgrade, filter cache showed a dramatic change
and CPU is much more optimized as compared to before. As SignalFx experiences a scale
growth and happily serves the increasing indexing and query load, the team is much more
confident about the Elasticsearch cluster performance.

Copyright 2017 © SignalFx, Inc Upgrading Elasticsearch With Zero Downtime | 20