International Journal on Recent and Innovation Trends in Computing and Communication ISSN: 2321-8169

Volume: 5 Issue: 11 222 – 229

_______________________________________________________________________________________________

A Review of Elastic Search: Performance Metrics and challenges

Subhani Shaik1 Nallamothu Naga Malleswara Rao2

Research scholar, Department of CSE, Professor, Department of IT,
Acharya Nagarjuna University ,, RVR & JC College of Engineering,
Guntur, A.P, India. Chowdavaram, Guntur, A.P,India.

Abstract: The most important aspect of a search engine is the search. Elastic search is a highly scalable search engine that stores data in a
structure, optimized for language based searches. When it comes to using Elastic search, there are lots of metrics engendered. By using Elastic
search to index millions of code repositories as well as indexing critical event data, you can satisfy the search needs of millions of users while
instantaneously providing strategic operational visions that help you iteratively improve customer service. In this paper we are going to study
about Elastic searchperformance metrics to watch, important Elastic search challenges, and how to deal with them. This should be helpful to
anyone new to Elastic search, and also to experienced users who want a quick start into performance monitoring of Elastic search.

Keywords: Elastic search, Query latency, Index flush, Garbage collection, JVM metrics, Cache metrics.
__________________________________________________*****_________________________________________________
node is also able to function as a data node. In order to
1. INTRODUCTION: improve reliability in larger clusters, users may launch
dedicated master-eligible nodes that do not store any data.
Elastic search is a highly scalable, distributed, open source
RESTful search and analytics engine. It is multitenant-capable a. Data nodes
with an HTTP web interface and schema-free JSON Every node that stores data in the form of index and performs
documents. Based on Apache Lucene, Elastic search is one of actions related to indexing, searching, and aggregating data is
the most popular enterprise search engines today and is a data node. In larger clusters, you may choose to create
capable of solving a growing number of use cases like log dedicated data nodes by adding node.master: false to the
analytics, real-time application monitoring, and click stream config file, ensuring that these nodes have enough resources to
analytics. Developed by Shay Banon and released in 2010, it handle data-related requests without the additional workload
relies heavily on Apache Lucene, a full-text search engine of cluster-related administrative tasks.
written in Java.Elastic search represents data in the form of
structured JSON documents, and makes full-text search b. Client nodes
accessible via RESTful API and web clients for languages like Client nodeis designed to act as a load balancer that helps
PHP, Python, and Ruby. It’s also elastic in the sense that it’s route indexing and search requests. Client nodes help to bear
easy to scale horizontally—simply add more nodes to some of the search workload so that data and master-eligible
distribute the load. Today, many companies, including nodes can focus on their core tasks.
Wikipedia, eBay, GitHub, and Datadog, use it to store, search,
and analyze large amounts of data on the fly.

2. ELASTICSEARCH-THEBASIC ELEMENTS
In Elastic search, a cluster is made up of one or more
nodes.Each node is a single running instance of Elastic search,
and its elasticsearch.yml configuration file designates which
cluster it belongs to (cluster.name) and what type of node it
can be. Any property, including cluster name set in the
configuration file can also be specified via command line
argument. The three most common types of nodes in Elastic
search are:

2.1 Master-eligible nodes

Every node in Elastic search is master-eligible by default
unless otherwise specified. Each cluster automatically elects a
master node from all of the master-eligible nodes. The master
node is responsible for coordinating cluster tasks like
distributing shards across nodes, and creating and deleting
indices. If the current master node experiences a failure
master-eligible nodes elect a new master. Any master-eligible Fig:1 Elastic Search Cluster
222
IJRITCC | November 2017, Available @ http://www.ijritcc.org
_______________________________________________________________________________________
 International Journal on Recent and Innovation Trends in Computing and Communication ISSN: 2321-8169
Volume: 5 Issue: 11 222 – 229
_______________________________________________________________________________________________
3. ELASTICSEARCHDATA ORGANIZATION
In Elasticsearch, interrelated data is stored in the same index
thatcontains a set of related documents in JSON format.
Elasticsearch’s secret sauce for full-text search is Lucene’s
inverted index. When a document is indexed, Elasticsearch
automatically creates an inverted index for each field; the
inverted index maps terms to the documents that contain those
terms.An index is stored across one or more primary shards,
and zero or more replica shards, and each shard is a complete
instance of Lucene, like a mini search engine.
When creating an index, we can specify the number of primary
shards, as well as the number of replicas per primary. The
defaults are five primary shards per index, and one replica per
primary. The number of primary shards cannot be changed
once an index has been created. The number of replicas can be
Figure 3: Processing of Search Request
updated later on as needed. To protect against data loss, the
master node ensures that each replica shard is not allocated to
If search is a customer-facing feature you should monitor
the same node as its primary shard.
query latency and take action if it surpasses a threshold. It’s
important to monitor relevant metrics about queries and
fetches that can help you determine how your searches
perform over time. For example, you may want to track spikes
and long-term increases in query requests, so that you can be
prepared to tweak your configuration to optimize for better
performance and reliability.


Search performance metrics
Query load: Monitoring the number of queries currently in
progress can give you a rough idea of how many requests your
cluster is dealing with at any particular moment in time.
Consider alerting on unusual spikes or dips that may point to
Fig:2 Elastic search data rganization underlying problems. You may also want to monitor the size
4. of the search thread pool queue.


4. ELASTIC SEARCH PERFORMANCE
METMETRICS : Query latency: Though Elasticsearch does not explicitly
Elasticsearch provides plenty of metrics to detect problems provide this metric, monitoring tools can help you use the
like unreliable nodes, out-of-memory errors, and long garbage available metrics to calculate the average query latency by
collection times. All these metrics are accessible via sampling the total number of queries and the total elapsed time
Elasticsearch’s API as well as single-purpose monitoring tools at regular intervals. Set an alert if latency exceeds a threshold,
like Elastic’s Marvel and universal monitoring services like and if it fires, look for potential resource bottlenecks, or
Datadog. investigate whether you need to optimize your queries.

4.1Search and indexing performance  Fetch latency: The fetch phase, should typically take much
In Elasticsearch we have two types of requests, the search less time than the query phase. If this metric isconstantly
requests and index requests which aresimilar to read and write increasing, this could indicate a problem with slow
requests in a traditional database system. disks, enriching of documents (highlighting relevant text in
search results, etc.), or requesting too many results.

 Client sends a search request to Node 2

4.1.1 Search Request:

 The coordinating node, Node 2 sends the query to a

4.1.2 Index Requests
Indexing requests are similar to write requests in a traditional

 Each shard executes the query locally and delivers

copy of every shard in the index. database system. If your Elasticsearch workload is write-
heavy, it’s important to monitor and analyze how effectively
results to Node 2. Node 2 sorts and compiles them you are able to update indices with new information. When
new information is added to an index, or existing information
 Node 2 finds out which documents need to be fetched
into a global priority queue.
is updated or deleted, each shard in the index is updated via
and sends a multi GET request to the relevant two processes: refresh and flush.

 Each shard loads the documents and returns them to  Index refresh
shards.5.

Newly indexed documents are not immediately made available

 Node 2 delivers the search results to the client.
Node 2.
for search. First they are written to an in-memory buffer where
they await the next index refresh, which occurs once per
223
IJRITCC | November 2017, Available @ http://www.ijritcc.org
_______________________________________________________________________________________
International Journal on Recent and Innovation Trends in Computing and Communication ISSN: 2321-8169
Volume: 5 Issue: 11 222 – 229
_______________________________________________________________________________________________
second by default. The refresh process creates a new in-
memory segment from the contents of the in-memory buffer
(making the newly indexed documents searchable), then
empties the buffer, as shown below.

Figure: 5The index flush process

Indexing Performance Metrics

Elasticsearch provides a number of metrics to assess indexing

 Indexing latency: Monitoring tools can help to

performance and to optimize update your indices.

calculate the average indexing latency from the

Figure 4. The index refresh process
available index_total and index_time_in_millis metric
s. If the latency is increasing, user is trying to index
Shards of an index are composed of multiple segments. The
too many documents at one time.To index a lot of
core data structure from Lucene, a segment is essentially a
documents without new information to be
change set for the index. These segments are created with
immediately available for search, you can optimize
every refresh and subsequently merged together over time in
for indexing performance over search performance by
the background to ensure efficient use of resources. Each
decreasing refresh frequency until you are done
segment uses file handles, memory, and CPU. Segments are
indexing.
mini-inverted indices that map terms to the documents that

contain those terms. Every time an index is searched, a
Flush latency: Because data is not persisted to disk
primary or replica version of each shard must be searched by,
until a flush is successfully completed, it can be
in turn, searching every segment in that shard.
useful to track flush latency and take action if
 writing the information to a new segment during the
A segment is immutable, so updating a document means:
performance begins to take a dive. If this metric is
increasing steadily, it could indicate a problem with
 marking the old information as deleted
refresh process
slow disks; this problem may escalate and eventually
prevent from being able to add new information to
The old information is eventually deleted when the outdated index.
segment is merged with another segment.
4.2 Memory usage and garbage collection
Index flush Memory is one of the key resources when running
When the newly indexed documents are added to the in- Elasticsearch. Elasticsearch and Lucene utilize all of the
memory buffer, they are also appended to the shard’s translog: available RAM on your nodes in two ways: JVM heap and the
a persistent, write-ahead transaction log of operations. file system cache. Elasticsearch runs in the Java Virtual
Whenever the translog reaches a maximum size which is Machine (JVM), which means that JVM garbage collection
512MB by default, a flush is triggered. During a flush, any duration and frequency will be other important areas to
documents in the in-memory buffer are refreshed (stored on monitor.
new segments), all in-memory segments are committed to
disk, and the translog is cleared. JVM heap:
The translog helps prevent data loss in the event that a node Elasticsearch stresses the importance of a JVM heap size. In
fails. It is designed to help a shard recover operations that may general, Elasticsearch’s rule of thumb is allocating less than 50
otherwise have been lost between flushes. The log is percent of available RAM to JVM heap, and never going
committed to disk every five seconds, or upon each successful higher than 32 GB.
index, delete, update, or bulk request, whichever occurs first. The less heap memory you allocate to Elasticsearch, the more
RAM remains available for Lucene, which relies heavily on
the file system cache to serve requests quickly. If the heap size
is too small we may get out-of-memory errors or reduced
throughput as the application faces constant short pauses from
frequent garbage collections. Elasticsearch’s default
installation sets a JVM heap size of 1 gigabyte, which is too
small for most use cases. The other option is to set the JVM
heap size (with equal minimum and maximum sizes to prevent
224
IJRITCC | November 2017, Available @ http://www.ijritcc.org
_______________________________________________________________________________________
International Journal on Recent and Innovation Trends in Computing and Communication ISSN: 2321-8169
Volume: 5 Issue: 11 222 – 229
_______________________________________________________________________________________________
the heap from resizing) on the command line every time you 4.3 Host-level network and system metrics
start up Elasticsearch: Host metrics to alert on
Disk space: This metric is significant if Elasticsearch cluster
Garbage collection is write-heavy. To insert or update anything we need a
Elasticsearch relies on garbage collection processes to free up sufficient disk space otherwise the node will fail. If less than
heap memory. Because garbage collection uses resources in 20 percent is available on a node, use a tool like Curator to
order to free up resources, you need to adjust the heap size. delete certain indices residing on that node that are taking up
Setting the heap too large can result in long garbage collection too much valuable disk space. Other alternative is to add more
times; these excessive pauses are dangerous because they can nodes, and let the master take care of automatically
lead your cluster to mistakenly register your node as having redistributing shards across the new nodes.
dropped off the grid.
Host metrics to watch
JVM metrics I/O utilization: Elasticsearch does a lot of writing to and
JVM heap in use: Elasticsearch is set up to initiate garbage reading from disk when segments are created, queried, and
collections whenever JVM heap usage hits 75 percent. As merged, For write-heavy clusters with nodes that are
shown above, it may be useful to monitor which nodes exhibit frequently experiencing heavy I/O activity, Elasticsearch
high heap usage, and set up an alert to find out if any node is recommends using SSDs to boost performance.
consistently using over 85 percent of heap memory; this
indicates that the rate of garbage collection isn’t keeping up CPU Utilization: Increase in CPU usage is usually caused by
with the rate of garbage creation. To address this problem, you a heavy search or indexing workload. Set up a notification to
can either increase your heap size (as long as it remains below find out if your nodes’ CPU usage is consistently increasing,
the recommended guidelines stated above), or scale out the and add more nodes to redistribute the load if needed.
cluster by adding more nodes.
Network bytes sent/received: Communication between nodes
JVM heap used vs. JVM heap committed: It can be helpful is a key component of a balanced cluster. Elasticsearch
to get an idea of how much JVM heap is currently in use, provides transport metrics about cluster communication.
compared to committed memory (the amount that Open file descriptors: File descriptors are used for node-to-
is guaranteed to be available). The amount of heap memory in node communications, client connections, and file operations.
use will typically take on a sawtooth pattern that rises when If this number reaches your system’s max capacity, then new
garbage accumulates and dips when garbage is collected. If the connections and file operations will not be possible until old
pattern starts to skew upward over time, this means that the ones have closed.
rate of garbage collection is not keeping up with the rate of
object creation, which could lead to slow garbage collection HTTP connections
times and, eventually, OutOfMemoryErrors. Requests sent in any language but Java will communicate with
Elastic search using RESTful API over HTTP. If the total
Garbage collection duration and frequency: Both young- number of opened HTTP connections is constantly increasing,
and old-generation garbage collectors undergo ―stop the it could indicate that your HTTP clients are not properly
world‖ phases, as the JVM halts execution of the program to establishing persistent connections. Reestablishing
collect dead objects. During this time, the node cannot connections adds extra milliseconds or even seconds to your
complete any tasks. Because the master node checks the status request response time. Make sure your clients are configured
of every other node every 30 seconds, if any node’s garbage properly to avoid negative impact on performance, or use one
collection time exceed 30 seconds, it will lead the master to of the official Elasticsearch clients, which already properly
believe that the node has failed. configure HTTP connections.

 Cluster status:
Memory usage 4.4 Cluster health and node availability
Elasticsearch makes excellent use of any RAM that has not
been allocated to JVM heap. Elasticsearch was designed to If the cluster status is yellow, at least one replica shard is
rely on the operating system’s file system cache to serve unallocated or missing. Search results will still be
requests quickly and reliably.A number of variables determine complete, but if more shards disappear, you may lose
whether or not Elasticsearch successfully reads from the file data.If the cluster status is red, at least one primary shard
system cache. If the segment file was recently written to disk is missing, and you are missing data, which means that
by Elasticsearch, it is already in the cache. However, if a node searches will return partial results. You will also be
has been shut off and rebooted, the first time a segment is blocked from indexing into that shard. Consider setting up
queried, the information will most likely have to be read from an alert to trigger if status has been yellow for more than 5
disk. This is one reason why it’s important to make sure your
 Initializing and unassigned shards
min or if the status has been red for the past minute.
cluster remains stable and that nodes do not crash.Generally,
it’s very important to monitor memory usage on your nodes, When you first create an index, or when a node is
and give Elasticsearch as much RAM as possible, so it can rebooted, its shards will briefly be in an ―initializing‖ state
leverage the speed of the file system cache without running before transitioning to a status of ―started‖ or
out of space. ―unassigned‖, as the master node attempts to assign
shards to nodes in the cluster. If shards remain in an
225
IJRITCC | November 2017, Available @ http://www.ijritcc.org
_______________________________________________________________________________________
International Journal on Recent and Innovation Trends in Computing and Communication ISSN: 2321-8169
Volume: 5 Issue: 11 222 – 229
_______________________________________________________________________________________________
initializing or unassigned state too long, it could be a time. If caches hog too much of the heap, they may slow
warning sign that the cluster is unstable. things down instead of speeding them up.
In Elastic search, each field in a document can be stored in one
4.5 Resource saturation and errors of two forms: as an exact value or as full text. An exact value,
Elasticsearch nodes use thread pools to manage how threads such as a timestamp or a year, is stored exactly the way it was
consume memory and CPU. Since thread pool settings are indexed because you do not expect to receive to query 1/1/16
automatically configured based on the number of processors, it as ―January 1st, 2016.‖ If a field is stored as full text, that
usually doesn’t make sense to tweak them. If the nodes are not means it is analyzed—basically, it is broken down into tokens,
able to keep up we can add more nodes to handle all of the and, depending on the type of analyzer, punctuation and stop
concurrent requests. Fielddata and filter cache usage is another words like ―is‖ or ―the‖ may be removed. The analyzer
area to monitor, as evictions may point to inefficient queries or converts the field into a normalized format that enables it to
signs of memory pressure. match a wider range of queries.
Elastic search uses two main types of caches to serve search
Thread pool queue and rejections requests more quickly: fielddata and filter.
Each node maintains many types of thread pools; The most
important nodes to monitor are search, index, merge, and Fielddata cache
bulk.The size of each thread pool’s queue represents how The fielddata cache is used when sorting or aggregating on a
many requests are waiting to be served while the node is field, a process that basically has to uninvent the inverted
currently at capacity. The queue allows the node to track and index to create an array of every field value per field, in
eventually serve these requests instead of discarding them. document order.
Thread pool rejections arise once the thread pool’s maximum
queue size is reached. Filter cache
Filter caches also use JVM heap. Elastic search automatically
Metrics to watch cached filtered queries with a max value of 10 percent of the
Thread pool queues : Large queues are not ideal because they heap, and evicted the least recently used data. Elastic search
use up resources and also increase the risk of losing requests if automatically began optimizing its filter cache, based on
a node goes down. If you see the number of queued and frequency and segment size (caching only occurs on segments
rejected threads increasing steadily, you may want to try that have fewer than 10,000 documents or less than 3 percent
slowing down the rate of requests (if possible), increasing the of total documents in the index).
number of processors on your nodes, or increasing the number
of nodes in the cluster. As shown in the screenshot below, Cache metrics to watch
query load spikes correlate with spikes in search thread pool Fielddata cache evictions: Ideally, you want to limit the
queue size, as the node attempts to keep up with rate of query number of fielddata evictions because they are I/O intensive. If
requests. you’re seeing a lot of evictions and you cannot increase your
memory at the moment, Elastic search recommends a
temporary fix of limiting fielddata cache to 20 percent of heap;
Elastic search also recommends using doc values whenever
possible because they serve the same purpose as fielddata.
However, because they are stored on disk, they do not rely on
JVM heap. Although doc values cannot be used for analyzed
string fields, they do save field data usage when aggregating or
sorting on other types of fields.

Filter cache evictions: Each segment maintains its own

individual filter cache. Since evictions are costlier operations
on large segments than small segments, there’s no clear-cut
way to assess how serious each eviction may be. However, if
Fig:6.Thread pool queues
you see evictions occurring more often, this may indicate that
you are not using filters to your best advantage—you could
Bulk rejections and bulk queues: Bulk rejections are usually
just be creating new ones and evicting old ones on a frequent
related to trying to index too many documents in one bulk
basis, defeating the purpose of even using a cache.
request. Bulk operations are a more efficient way to send
many requests at one time. Generally, if you want to perform
Pending tasks
many actions like create an index, or add, update, or delete
Pending tasks such as creating indices and assigning shards to
documents, you should try to send the requests as a bulk
nodes can only be handled by master nodes. Pending tasks are
operation instead of many individual requests.
processed in priority order—urgent comes first, then high
priority. They start to accumulate when the number of changes
Cache usage metrics
occurs more quickly than the master can process them.The
Each query request is sent to every shard in an index, which
number of pending tasks indicates how smoothly a cluster is
then hits every segment of each of those shards. Elastic search
operating. If your master node is very busy and the number of
caches queries on a per-segment basis to speed up response
226
IJRITCC | November 2017, Available @ http://www.ijritcc.org
_______________________________________________________________________________________
International Journal on Recent and Innovation Trends in Computing and Communication ISSN: 2321-8169
Volume: 5 Issue: 11 222 – 229
_______________________________________________________________________________________________
pending tasks doesn’t subside, it can lead to an unstable of your Elastic search home directory) for a line similar to the
cluster. following:[TIMESTAMP] ... Cluster health status changed
from [GREEN] to [RED].
Unsuccessful GET requests Reasons for node failure can vary, ranging from hardware or
A GET request is more straightforward than a normal search hypervisor failures, to out-of-memory errors. If it is a
request—it retrieves a document based on its ID. An temporary failure, you can try to get the disconnected node(s)
unsuccessful get-by-ID request means that the document ID to recover and rejoin the cluster. If it is a permanent failure,
was not found. and you are not able to recover the node, you can add new
nodes and let Elastic search take care of recovering from any
available replica shards; replica shards can be promoted to
5. ELASTICSEARCH CHALLENGES primary shards and redistributed on the new nodes you just
Elasticsearch was intended to allow its users to get up and added.
running quickly, without having to understand all of its inner However, if you lost both the primary and replica copy of a
workings. shard, you can try to recover as much of the missing data as
possible by using Elastic search snapshot and restore module.
5.1 Cluster status. If you’re not already familiar with this module, it can be used
Cluster status is reported as red if one or more primary shards to store snapshots of indices over time in a remote repository
and its replicas is missing, and yellow if one or more replica for backup purposes.
shards is missing. Normally, this happens when a node drops
off the cluster for hardware failure, long garbage collection 5.2Disk space
time, etc. Once the node recovers, its shards will remain in an If all data nodes are running low on disk space, add more data
initializing state before they transition back to active status. nodes to a cluster. Make sure that indices have enough primary
The number of initializing shards typically peaks when a node shards to be able to balance their data across all those
rejoins the cluster, and then drops back down as the shards nodes.However, if only certain nodes are running out of disk
transition into an active state. space means an index is initialized with too few shards. It is
hard for Elastic search to distribute these shards across nodes
in a balanced manner.Elastic search takes available disk space
into account when allocating shards to nodes. By default, it
will not assign shards to nodes that have over 85 percent disk
in use. In Datadog, you can set up a threshold alert to notify
you when any individual data node’s disk space usage
approaches 80 percent, which should give you enough time to
take action.

 One is to remove outdated data and store it off the

There are two remedies for low disk space.
Fig: 7.Cluster status
During this initialization period, your cluster state may
 Second is storing all of your data on the cluster
cluster.
transition from green to yellow or red until the shards on the
recovering node regain active status. In many cases, a brief vertically or horizontally.
status change to yellow or red may not require any action on
your part. 5.3 Execution time of Searches
However, if you notice that your cluster status is lingering in Search performance varies according to the type of data that is
red or yellow state for an extended period of time, verify that being searched and how each query is structured. Depending
the cluster is recognizing the correct number of Elastic search on the way data is organized, to speed up search performance,
nodes, either by consulting Data dog’s dashboard or by
 Custom routing allows you to store related data on the
we have two methods custom routing andforce merging.
querying the Cluster Health API
same shard, so that you only have to search a single shard
to satisfy a query.In Elasticsearch, every search request
has to check every segment of each shard it hits. So once
you have reduced the number of shards you’ll have to
search, you can also reduce the number of segments per
shard by triggering the Force Merge API on one or more
of your indices. The Force Merge API prompts the
segments in the index to continue merging until each
shard’s segment count is reduced

 Force mergingwhen it comes to shards with a large

Fig:8. The Cluster Health API to max_num_segments which is 1, by default.
If the number of active nodes is lower than expected, it means
that at least one of your nodes lost its connection and hasn’t number of segments, the force merge process becomes
been able to rejoin the cluster. To find out which node(s) left much more computationally expensive. Force merging an
the cluster, check the logs (located by default in the logs folder index of 10,000 segments down to 5,000 segments doesn’t

227
IJRITCC | November 2017, Available @ http://www.ijritcc.org
_______________________________________________________________________________________
International Journal on Recent and Innovation Trends in Computing and Communication ISSN: 2321-8169
Volume: 5 Issue: 11 222 – 229
_______________________________________________________________________________________________
take much time, but merging 10,000 segments all the way settings. With this in place, the index will only
down to one segment can take hours. The more merging commit writes to disk upon every sync_interval,
that must occur, the more resources you take away from rather than after each request, leaving more of its
fulfilling search requests, which may defeat the purpose of resources free to serve indexing requests.
calling a force merge in the first place. It is a good idea to
schedule a force merging during non-peak hours, such as 5.5Bulk thread pool rejections
overnight, when you don’t expect many search or Thread pool rejections are typically a sign that you are sending
indexing requests. too many requests to your nodes, too quickly. If this is a
temporary situation you can try to slow down the rate of your
5.4Index-heavy workload. requests. However, if you want your cluster to be able to
Elastic search comes pre-configured with many settings to sustain the current rate of requests, you will probably need to
retain enough resources for searching and indexing data. scale out your cluster by adding more data nodes. In order to
However, if the usage of Elastic search is heavily skewed utilize the processing power of the increased number of nodes,
towards writes, it makes sense to tweak certain settings to you should also make sure that your indices contain enough
boost indexing performance, even if it means losing some shards to be able to spread the load evenly across all of your
search performance or data replication. nodes.

 Shard allocation: If you are creating an index to

Methods to optimize use case for indexing.
6. CONCLUSION
update frequently, allocate one primary shard per Elastic search lets you make amazing things quite easily. It
node in a cluster, and two or more primary shards per provides great features at great speeds and scale.In this paper,
node, but only if you have a lot of CPU and disk we’ve covered important areas of Elastic searchsuch as Search
bandwidth on those nodes. However, shard and indexing performance, Memory and garbage collection,
overallocation adds overhead and may negatively Host-level system and network metrics, Cluster health and
impact search, since search requests need to hit every node availability and Resource saturation and errors.Elastic
shard in the index. If you assign fewer primary shards search metrics along with node-level system metrics will
than the number of nodes, you may create hotspots, discover which areas are the most meaningful for specific use
as the nodes that contain those shards will need to case.
handle more indexing requests than nodes that don’t
contain any of the index’s shards.
 Disable merge throttling: Merge throttling is
REFERENCES
[1] https://curatedsql.com/2016/09/29/monitoring-
Elasticsearch’s automatic tendency to throttle elasticsearch-performance/
indexing requests when it detects that merging is [2] https://blog.codecentric.de/en/2014/05/elasticsearch-
falling behind indexing. Update cluster settings to indexing-performance-cheatsheet/
disable merge throttling to optimize indexing [3] https://sematext.com/publications/performance-monitoring-

 Increase the size of the indexing buffer:

performance, not search. essentials-elasticsearch-edition.pdf
[4] https://www.datadoghq.com/blog/elasticsearch-
This(indices.memory.index_buffer_size) setting performance-scaling-problems/
determines how full the buffer can get before its [5] https://dzone.com/articles/top-10-elasticsearch-metrics
documents are written to a segment on disk. The [6] Elastic search: Guide – https://www.elastic.co/guide
default setting limits this value to 10 percent of the [7] Elasticsearch: Issues –
total heap in order to reserve more of the heap for https://github.com/elasticsearch/elasticsearch/issues
serving search requests, which doesn’t help you if [8] Heroku postgres production tier technical characterization,
you’re using Elastic search primarily for indexing.
 Index first, replicate later: When you initialize an
2013 – https://devcenter.heroku.com/articles/heroku-
postgres-production-tier-technical-characterization
index, specify zero replica shards in the index [9] PostgreSQL: PostgreSQL documentation, 2013 –
settings, and add replicas after you’re done indexing. http://www.postgresql.org/docs/current/static/
This will boost indexing performance, but it can be a [10] Szegedi, Attila: Everything i ever learned about jVM
bit risky if the node holding the only copy of the data performance tuning @twitter

 Refresh less frequently: Increase the refresh interval

crashes before you have a chance to replicate it.
About the Authors
in the Index Settings API. By default, the index
refresh process occurs every second, but during heavy
Mr. Subhani Shaik is working as Assistant
indexing periods, reducing the refresh frequency can professor in Department of computer science and

 Tweak your translog settings: Elastic search

help alleviate some of the workload. Engineering at St. Mary’s group of institutions
Guntur, he has 12 years of TeachingExperience in
will flush translog data to disk after every request, the academics.
reducing the risk of data loss in the event of hardware
failure. If you want to prioritize indexing
performance over potential data loss, you can
change index.translogdurability to asyncin the index
228
IJRITCC | November 2017, Available @ http://www.ijritcc.org
_______________________________________________________________________________________
International Journal on Recent and Innovation Trends in Computing and Communication ISSN: 2321-8169
Volume: 5 Issue: 11 222 – 229
_______________________________________________________________________________________________
Dr. Nallamothu Naga Malleswara Rao is working
as Professor in the Department of Information
Technology at RVR & JC College of Engineering
with 25 years of Teaching Experience in the
academics.

229
IJRITCC | November 2017, Available @ http://www.ijritcc.org
_______________________________________________________________________________________