curl \
--request DELETE 'http://api.example.com/_inference/{inference_id}' \
--header "Authorization: $API_KEY"
Get an autoscaling policy
Generally available; Added in 7.11.0
NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.
curl \
--request GET 'http://api.example.com/_autoscaling/policy/{name}' \
--header "Authorization: $API_KEY"
{
"roles": <roles>,
"deciders": <deciders>
}
Create or update an autoscaling policy
Generally available; Added in 7.11.0
NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
.
Body
Required
-
Decider settings.
External documentation
curl \
--request PUT 'http://api.example.com/_autoscaling/policy/{name}' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"roles\": [],\n \"deciders\": {\n \"fixed\": {\n }\n }\n}"'
{
"roles": [],
"deciders": {
"fixed": {
}
}
}
{
"roles" : [ "data_hot" ],
"deciders": {
"fixed": {
}
}
}
{
"acknowledged": true
}
curl \
--request DELETE 'http://api.example.com/_application/analytics/{name}' \
--header "Authorization: $API_KEY"
Path parameters
-
A comma-separated list of node identifiers or names used to limit the returned information.
Query parameters
-
The unit used to display byte values.
Values are
b
,kb
,mb
,gb
,tb
, orpb
. -
List of columns to appear in the response. Supports simple wildcards.
-
List of columns that determine how the table should be sorted. Sorting defaults to ascending and can be changed by setting
:asc
or:desc
as a suffix to the column name. -
If
true
, the request computes the list of selected nodes from the local cluster state. Iffalse
the list of selected nodes are computed from the cluster state of the master node. In both cases the coordinating node will send requests for further information to each selected node. -
Period to wait for a connection to the master node.
Values are
-1
or0
.
curl \
--request GET 'http://api.example.com/_cat/allocation/{node_id}' \
--header "Authorization: $API_KEY"
[
{
"shards": "1",
"shards.undesired": "0",
"write_load.forecast": "0.0",
"disk.indices.forecast": "260b",
"disk.indices": "260b",
"disk.used": "47.3gb",
"disk.avail": "43.4gb",
"disk.total": "100.7gb",
"disk.percent": "46",
"host": "127.0.0.1",
"ip": "127.0.0.1",
"node": "CSUXak2",
"node.role": "himrst"
}
]
Get the cluster health status
Generally available
IMPORTANT: CAT APIs are only intended for human consumption using the command line or Kibana console.
They are not intended for use by applications. For application consumption, use the cluster health API.
This API is often used to check malfunctioning clusters.
To help you track cluster health alongside log files and alerting systems, the API returns timestamps in two formats:
HH:MM:SS
, which is human-readable but includes no date information;
Unix epoch time
, which is machine-sortable and includes date information.
The latter format is useful for cluster recoveries that take multiple days.
You can use the cat health API to verify cluster health across multiple nodes.
You also can use the API to track the recovery of a large cluster over a longer period of time.
Required authorization
- Cluster privileges:
monitor
Query parameters
-
The unit used to display time values.
Values are
nanos
,micros
,ms
,s
,m
,h
, ord
. -
If true, returns
HH:MM:SS
and Unix epoch timestamps. -
List of columns to appear in the response. Supports simple wildcards.
-
List of columns that determine how the table should be sorted. Sorting defaults to ascending and can be changed by setting
:asc
or:desc
as a suffix to the column name.
curl \
--request GET 'http://api.example.com/_cat/health' \
--header "Authorization: $API_KEY"
[
{
"epoch": "1475871424",
"timestamp": "16:17:04",
"cluster": "elasticsearch",
"status": "green",
"node.total": "1",
"node.data": "1",
"shards": "1",
"pri": "1",
"relo": "0",
"init": "0",
"unassign": "0",
"unassign.pri": "0",
"pending_tasks": "0",
"max_task_wait_time": "-",
"active_shards_percent": "100.0%"
}
]
Get datafeeds
Generally available; Added in 7.7.0
Get configuration and usage information about datafeeds.
This API returns a maximum of 10,000 datafeeds.
If the Elasticsearch security features are enabled, you must have monitor_ml
, monitor
, manage_ml
, or manage
cluster privileges to use this API.
IMPORTANT: CAT APIs are only intended for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, use the get datafeed statistics API.
Required authorization
- Cluster privileges:
monitor_ml
Query parameters
-
Specifies what to do when the request:
- Contains wildcard expressions and there are no datafeeds that match.
- Contains the
_all
string or no identifiers and there are no matches. - Contains wildcard expressions and there are only partial matches.
If
true
, the API returns an empty datafeeds array when there are no matches and the subset of results when there are partial matches. Iffalse
, the API returns a 404 status code when there are no matches or only partial matches. -
Comma-separated list of column names to display.
-
Comma-separated list of column names or column aliases used to sort the response.
-
The unit used to display time values.
Values are
nanos
,micros
,ms
,s
,m
,h
, ord
.
curl \
--request GET 'http://api.example.com/_cat/ml/datafeeds' \
--header "Authorization: $API_KEY"
[
{
"id": "datafeed-high_sum_total_sales",
"state": "stopped",
"buckets.count": "743",
"search.count": "7"
},
{
"id": "datafeed-low_request_rate",
"state": "stopped",
"buckets.count": "1457",
"search.count": "3"
},
{
"id": "datafeed-response_code_rates",
"state": "stopped",
"buckets.count": "1460",
"search.count": "18"
},
{
"id": "datafeed-url_scanning",
"state": "stopped",
"buckets.count": "1460",
"search.count": "18"
}
]
Get anomaly detection jobs
Generally available; Added in 7.7.0
Get configuration and usage information for anomaly detection jobs.
This API returns a maximum of 10,000 jobs.
If the Elasticsearch security features are enabled, you must have monitor_ml
,
monitor
, manage_ml
, or manage
cluster privileges to use this API.
IMPORTANT: CAT APIs are only intended for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, use the get anomaly detection job statistics API.
Required authorization
- Cluster privileges:
monitor_ml
Query parameters
-
Specifies what to do when the request:
- Contains wildcard expressions and there are no jobs that match.
- Contains the
_all
string or no identifiers and there are no matches. - Contains wildcard expressions and there are only partial matches.
If
true
, the API returns an empty jobs array when there are no matches and the subset of results when there are partial matches. Iffalse
, the API returns a 404 status code when there are no matches or only partial matches. -
The unit used to display byte values.
Values are
b
,kb
,mb
,gb
,tb
, orpb
. -
Comma-separated list of column names to display.
-
Comma-separated list of column names or column aliases used to sort the response.
-
The unit used to display time values.
Values are
nanos
,micros
,ms
,s
,m
,h
, ord
.
curl \
--request GET 'http://api.example.com/_cat/ml/anomaly_detectors' \
--header "Authorization: $API_KEY"
[
{
"id": "high_sum_total_sales",
"s": "closed",
"dpr": "14022",
"mb": "1.5mb"
},
{
"id": "low_request_rate",
"s": "closed",
"dpr": "1216",
"mb": "40.5kb"
},
{
"id": "response_code_rates",
"s": "closed",
"dpr": "28146",
"mb": "132.7kb"
},
{
"id": "url_scanning",
"s": "closed",
"dpr": "28146",
"mb": "501.6kb"
}
]
Get snapshot information
Generally available; Added in 2.1.0
Get information about the snapshots stored in one or more repositories. A snapshot is a backup of an index or running Elasticsearch cluster. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the get snapshot API.
Required authorization
- Cluster privileges:
monitor_snapshot
Path parameters
-
A comma-separated list of snapshot repositories used to limit the request. Accepts wildcard expressions.
_all
returns all repositories. If any repository fails during the request, Elasticsearch returns an error.
Query parameters
-
List of columns to appear in the response. Supports simple wildcards.
-
List of columns that determine how the table should be sorted. Sorting defaults to ascending and can be changed by setting
:asc
or:desc
as a suffix to the column name. -
Period to wait for a connection to the master node.
Values are
-1
or0
. -
Unit used to display time values.
Values are
nanos
,micros
,ms
,s
,m
,h
, ord
.
curl \
--request GET 'http://api.example.com/_cat/snapshots/{repository}' \
--header "Authorization: $API_KEY"
[
{
"id": "snap1",
"repository": "repo1",
"status": "FAILED",
"start_epoch": "1445616705",
"start_time": "18:11:45",
"end_epoch": "1445616978",
"end_time": "18:16:18",
"duration": "4.6m",
"indices": "1",
"successful_shards": "4",
"failed_shards": "1",
"total_shards": "5"
},
{
"id": "snap2",
"repository": "repo1",
"status": "SUCCESS",
"start_epoch": "1445634298",
"start_time": "23:04:58",
"end_epoch": "1445634672",
"end_time": "23:11:12",
"duration": "6.2m",
"indices": "2",
"successful_shards": "10",
"failed_shards": "0",
"total_shards": "10"
}
]
Explain the shard allocations
Generally available; Added in 5.0.0
Get explanations for shard allocations in the cluster. For unassigned shards, it provides an explanation for why the shard is unassigned. For assigned shards, it provides an explanation for why the shard is remaining on its current node and has not moved or rebalanced to another node. This API can be very useful when attempting to diagnose why a shard is unassigned or why a shard continues to remain on its current node when you might expect otherwise.
Query parameters
-
If true, returns information about disk usage and shard sizes.
-
If true, returns YES decisions in explanation.
-
Period to wait for a connection to the master node.
Values are
-1
or0
.
curl \
--request GET 'http://api.example.com/_cluster/allocation/explain' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"index\": \"my-index-000001\",\n \"shard\": 0,\n \"primary\": false,\n \"current_node\": \"my-node\"\n}"'
{
"index": "my-index-000001",
"shard": 0,
"primary": false,
"current_node": "my-node"
}
{
"index" : "my-index-000001",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "INDEX_CREATED",
"at" : "2017-01-04T18:08:16.600Z",
"last_allocation_status" : "no"
},
"can_allocate" : "no",
"allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"node_allocation_decisions" : [
{
"node_id" : "8qt2rY-pT6KNZB3-hGfLnw",
"node_name" : "node-0",
"transport_address" : "127.0.0.1:9401",
"roles" : ["data", "data_cold", "data_content", "data_frozen", "data_hot", "data_warm", "ingest", "master", "ml", "remote_cluster_client", "transform"],
"node_attributes" : {},
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "filter",
"decision" : "NO",
"explanation" : "node does not match index setting [index.routing.allocation.include] filters [_name:\"nonexistent_node\"]"
}
]
}
]
}
{
"index" : "my-index-000001",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"at" : "2017-01-04T18:03:28.464Z",
"failed shard on node [mEKjwwzLT1yJVb8UxT6anw]: failed recovery, failure RecoveryFailedException",
"reason": "ALLOCATION_FAILED",
"failed_allocation_attempts": 5,
"last_allocation_status": "no",
},
"can_allocate": "no",
"allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "3sULLVJrRneSg0EfBB-2Ew",
"node_name" : "node_t0",
"transport_address" : "127.0.0.1:9400",
"roles" : ["data_content", "data_hot"],
"node_decision" : "no",
"store" : {
"matching_size" : "4.2kb",
"matching_size_in_bytes" : 4325
},
"deciders" : [
{
"decider": "max_retry",
"decision" : "NO",
"explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2024-07-30T21:04:12.166Z], failed_attempts[5], failed_nodes[[mEKjwwzLT1yJVb8UxT6anw]], delayed=false, details[failed shard on node [mEKjwwzLT1yJVb8UxT6anw]: failed recovery, failure RecoveryFailedException], allocation_status[deciders_no]]]"
}
]
}
]
}
Get cluster-wide settings
Generally available
By default, it returns only settings that have been explicitly defined.
Required authorization
- Cluster privileges:
monitor
Query parameters
-
If
true
, returns settings in flat format. -
If
true
, returns default cluster settings from the local node. -
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
.
curl \
--request GET 'http://api.example.com/_cluster/settings' \
--header "Authorization: $API_KEY"
Query parameters
-
If true, known idle threads (e.g. waiting in a socket select, or to get a task from an empty queue) are filtered out.
-
The interval to do the second sampling of threads.
Values are
-1
or0
. -
Number of samples of thread stacktrace.
-
Specifies the number of hot threads to provide information for.
-
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
The type to sample.
Values are
cpu
,wait
,block
,gpu
, ormem
. -
The sort order for 'cpu' type (default: total)
Values are
cpu
,wait
,block
,gpu
, ormem
.
curl \
--request GET 'http://api.example.com/_nodes/{node_id}/hot_threads' \
--header "Authorization: $API_KEY"
Query parameters
-
Comma-separated list or wildcard expressions of fields to include in fielddata and suggest statistics.
-
Comma-separated list or wildcard expressions of fields to include in fielddata statistics.
-
Comma-separated list or wildcard expressions of fields to include in the statistics.
-
Comma-separated list of search groups to include in the search statistics.
-
If true, the call reports the aggregated disk usage of each one of the Lucene index files (only applies if segment stats are requested).
-
Indicates whether statistics are aggregated at the cluster, index, or shard level.
Values are
cluster
,indices
, orshards
. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
A comma-separated list of document types for the indexing index metric.
-
If
true
, the response includes information from segments that are not loaded into memory.
curl \
--request GET 'http://api.example.com/_nodes/stats' \
--header "Authorization: $API_KEY"
curl \
--request GET 'http://api.example.com/_nodes/usage' \
--header "Authorization: $API_KEY"
Set a connector sync job error
Technical preview
Set the error
field for a connector sync job and set its status
to error
.
To sync data using self-managed connectors, you need to deploy the Elastic connector service on your own infrastructure. This service runs automatically on Elastic Cloud for Elastic managed connectors.
curl \
--request PUT 'http://api.example.com/_connector/_sync_job/{connector_sync_job_id}/_error' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"error\": \"some-error\"\n}"'
{
"error": "some-error"
}
Pause a follower
Generally available; Added in 6.5.0
Pause a cross-cluster replication follower index. The follower index will not fetch any additional operations from the leader index. You can resume following with the resume follower API. You can pause and resume a follower index to change the configuration of the following task.
Required authorization
- Cluster privileges:
manage_ccr
curl \
--request POST 'http://api.example.com/{index}/_ccr/pause_follow' \
--header "Authorization: $API_KEY"
{
"acknowledged" : true
}
Bulk index or delete documents
Generally available
Perform multiple index
, create
, delete
, and update
actions in a single request.
This reduces overhead and can greatly increase indexing speed.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:
- To use the
create
action, you must have thecreate_doc
,create
,index
, orwrite
index privilege. Data streams support only thecreate
action. - To use the
index
action, you must have thecreate
,index
, orwrite
index privilege. - To use the
delete
action, you must have thedelete
orwrite
index privilege. - To use the
update
action, you must have theindex
orwrite
index privilege. - To automatically create a data stream or index with a bulk API request, you must have the
auto_configure
,create_index
, ormanage
index privilege. - To make the result of a bulk operation visible to search using the
refresh
parameter, you must have themaintenance
ormanage
index privilege.
Automatic data stream creation requires a matching index template with data stream enabled.
The actions are specified in the request body using a newline delimited JSON (NDJSON) structure:
action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n
The index
and create
actions expect a source on the next line and have the same semantics as the op_type
parameter in the standard index API.
A create
action fails if a document with the same ID already exists in the target
An index
action adds or replaces a document as necessary.
NOTE: Data streams support only the create
action.
To update or delete a document in a data stream, you must target the backing index containing the document.
An update
action expects that the partial doc, upsert, and script and its options are specified on the next line.
A delete
action does not expect a source on the next line and has the same semantics as the standard delete API.
NOTE: The final line of data must end with a newline character (\n
).
Each newline character may be preceded by a carriage return (\r
).
When sending NDJSON data to the _bulk
endpoint, use a Content-Type
header of application/json
or application/x-ndjson
.
Because this format uses literal newline characters (\n
) as delimiters, make sure that the JSON actions and sources are not pretty printed.
If you provide a target in the request path, it is used for any actions that don't explicitly specify an _index
argument.
A note on the format: the idea here is to make processing as fast as possible.
As some of the actions are redirected to other shards on other nodes, only action_meta_data
is parsed on the receiving node side.
Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible.
There is no "correct" number of actions to perform in a single bulk request. Experiment with different settings to find the optimal size for your particular workload. Note that Elasticsearch limits the maximum size of a HTTP request to 100mb by default so clients must ensure that no request exceeds this size. It is not possible to index a single document that exceeds the size limit, so you must pre-process any such documents into smaller pieces before sending them to Elasticsearch. For instance, split documents into pages or chapters before indexing them, or store raw binary data in a system outside Elasticsearch and replace the raw data with a link to the external system in the documents that you send to Elasticsearch.
Client suppport for bulk requests
Some of the officially supported clients provide helpers to assist with bulk requests and reindexing:
- Go: Check out
esutil.BulkIndexer
- Perl: Check out
Search::Elasticsearch::Client::5_0::Bulk
andSearch::Elasticsearch::Client::5_0::Scroll
- Python: Check out
elasticsearch.helpers.*
- JavaScript: Check out
client.helpers.*
- .NET: Check out
BulkAllObservable
- PHP: Check out bulk indexing.
Submitting bulk requests with cURL
If you're providing text file input to curl
, you must use the --data-binary
flag instead of plain -d
.
The latter doesn't preserve newlines. For example:
$ cat requests
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
$ curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary "@requests"; echo
{"took":7, "errors": false, "items":[{"index":{"_index":"test","_id":"1","_version":1,"result":"created","forced_refresh":false}}]}
Optimistic concurrency control
Each index
and delete
action within a bulk API call may include the if_seq_no
and if_primary_term
parameters in their respective action and meta data lines.
The if_seq_no
and if_primary_term
parameters control how operations are run, based on the last modification to existing documents. See Optimistic concurrency control for more details.
Versioning
Each bulk item can include the version value using the version
field.
It automatically follows the behavior of the index or delete operation based on the _version
mapping.
It also support the version_type
.
Routing
Each bulk item can include the routing value using the routing
field.
It automatically follows the behavior of the index or delete operation based on the _routing
mapping.
NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing
setting enabled in the template.
Wait for active shards
When making bulk calls, you can set the wait_for_active_shards
parameter to require a minimum number of shard copies to be active before starting to process the bulk request.
Refresh
Control when the changes made by this request are visible to search.
NOTE: Only the shards that receive the bulk request will be affected by refresh.
Imagine a _bulk?refresh=wait_for
request with three documents in it that happen to be routed to different shards in an index with five shards.
The request will only wait for those three shards to refresh.
The other two shards that make up the index do not participate in the _bulk
request at all.
Query parameters
-
True or false if to include the document source in the error message in case of parsing errors.
-
If
true
, the response will include the ingest pipelines that were run for each index or create. -
The pipeline identifier to use to preprocess incoming documents. If the index has a default ingest pipeline specified, setting the value to
_none
turns off the default ingest pipeline for this request. If a final pipeline is configured, it will always run regardless of the value of this parameter. -
If
true
, Elasticsearch refreshes the affected shards to make this operation visible to search. Ifwait_for
, wait for a refresh to make this operation visible to search. Iffalse
, do nothing with refreshes. Valid values:true
,false
,wait_for
.Values are
true
,false
, orwait_for
. -
A custom value that is used to route operations to a specific shard.
-
Indicates whether to return the
_source
field (true
orfalse
) or contains a list of fields to return. -
A comma-separated list of source fields to exclude from the response. You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter. If the_source
parameter isfalse
, this parameter is ignored. -
A comma-separated list of source fields to include in the response. If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter. If the_source
parameter isfalse
, this parameter is ignored. -
The period each action waits for the following operations: automatic index creation, dynamic mapping updates, and waiting for active shards. The default is
1m
(one minute), which guarantees Elasticsearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur.Values are
-1
or0
. -
The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). The default is1
, which waits for each primary shard to be active.Values are
all
orindex-setting
. -
If
true
, the request's actions must target an index alias. -
If
true
, the request's actions must target a data stream (existing or to be created).
curl \
--request PUT 'http://api.example.com/_bulk' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{ \"index\" : { \"_index\" : \"test\", \"_id\" : \"1\" } }\n{ \"field1\" : \"value1\" }\n{ \"delete\" : { \"_index\" : \"test\", \"_id\" : \"2\" } }\n{ \"create\" : { \"_index\" : \"test\", \"_id\" : \"3\" } }\n{ \"field1\" : \"value3\" }\n{ \"update\" : {\"_id\" : \"1\", \"_index\" : \"test\"} }\n{ \"doc\" : {\"field2\" : \"value2\"} }"'
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }
{ "update" : {"_id" : "1", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"} }
{ "update" : { "_id" : "0", "_index" : "index1", "retry_on_conflict" : 3} }
{ "script" : { "source": "ctx._source.counter += params.param1", "lang" : "painless", "params" : {"param1" : 1}}, "upsert" : {"counter" : 1}}
{ "update" : {"_id" : "2", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"}, "doc_as_upsert" : true }
{ "update" : {"_id" : "3", "_index" : "index1", "_source" : true} }
{ "doc" : {"field" : "value"} }
{ "update" : {"_id" : "4", "_index" : "index1"} }
{ "doc" : {"field" : "value"}, "_source": true}
{ "update": {"_id": "5", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "update": {"_id": "6", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "create": {"_id": "7", "_index": "index1"} }
{ "my_field": "foo" }
{ "index" : { "_index" : "my_index", "_id" : "1", "dynamic_templates": {"work_location": "geo_point"}} }
{ "field" : "value1", "work_location": "41.12,-71.34", "raw_location": "41.12,-71.34"}
{ "create" : { "_index" : "my_index", "_id" : "2", "dynamic_templates": {"home_location": "geo_point"}} }
{ "field" : "value2", "home_location": "41.12,-71.34"}
{
"took": 30,
"errors": false,
"items": [
{
"index": {
"_index": "test",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 0,
"_primary_term": 1
}
},
{
"delete": {
"_index": "test",
"_id": "2",
"_version": 1,
"result": "not_found",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 404,
"_seq_no" : 1,
"_primary_term" : 2
}
},
{
"create": {
"_index": "test",
"_id": "3",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 2,
"_primary_term" : 3
}
},
{
"update": {
"_index": "test",
"_id": "1",
"_version": 2,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 200,
"_seq_no" : 3,
"_primary_term" : 4
}
}
]
}
{
"took": 486,
"errors": true,
"items": [
{
"update": {
"_index": "index1",
"_id": "5",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"_index": "index1",
"_id": "6",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"create": {
"_index": "index1",
"_id": "7",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1,
"status": 201
}
}
]
}
{
"items": [
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
}
]
}
Bulk index or delete documents
Generally available
Perform multiple index
, create
, delete
, and update
actions in a single request.
This reduces overhead and can greatly increase indexing speed.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:
- To use the
create
action, you must have thecreate_doc
,create
,index
, orwrite
index privilege. Data streams support only thecreate
action. - To use the
index
action, you must have thecreate
,index
, orwrite
index privilege. - To use the
delete
action, you must have thedelete
orwrite
index privilege. - To use the
update
action, you must have theindex
orwrite
index privilege. - To automatically create a data stream or index with a bulk API request, you must have the
auto_configure
,create_index
, ormanage
index privilege. - To make the result of a bulk operation visible to search using the
refresh
parameter, you must have themaintenance
ormanage
index privilege.
Automatic data stream creation requires a matching index template with data stream enabled.
The actions are specified in the request body using a newline delimited JSON (NDJSON) structure:
action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n
The index
and create
actions expect a source on the next line and have the same semantics as the op_type
parameter in the standard index API.
A create
action fails if a document with the same ID already exists in the target
An index
action adds or replaces a document as necessary.
NOTE: Data streams support only the create
action.
To update or delete a document in a data stream, you must target the backing index containing the document.
An update
action expects that the partial doc, upsert, and script and its options are specified on the next line.
A delete
action does not expect a source on the next line and has the same semantics as the standard delete API.
NOTE: The final line of data must end with a newline character (\n
).
Each newline character may be preceded by a carriage return (\r
).
When sending NDJSON data to the _bulk
endpoint, use a Content-Type
header of application/json
or application/x-ndjson
.
Because this format uses literal newline characters (\n
) as delimiters, make sure that the JSON actions and sources are not pretty printed.
If you provide a target in the request path, it is used for any actions that don't explicitly specify an _index
argument.
A note on the format: the idea here is to make processing as fast as possible.
As some of the actions are redirected to other shards on other nodes, only action_meta_data
is parsed on the receiving node side.
Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible.
There is no "correct" number of actions to perform in a single bulk request. Experiment with different settings to find the optimal size for your particular workload. Note that Elasticsearch limits the maximum size of a HTTP request to 100mb by default so clients must ensure that no request exceeds this size. It is not possible to index a single document that exceeds the size limit, so you must pre-process any such documents into smaller pieces before sending them to Elasticsearch. For instance, split documents into pages or chapters before indexing them, or store raw binary data in a system outside Elasticsearch and replace the raw data with a link to the external system in the documents that you send to Elasticsearch.
Client suppport for bulk requests
Some of the officially supported clients provide helpers to assist with bulk requests and reindexing:
- Go: Check out
esutil.BulkIndexer
- Perl: Check out
Search::Elasticsearch::Client::5_0::Bulk
andSearch::Elasticsearch::Client::5_0::Scroll
- Python: Check out
elasticsearch.helpers.*
- JavaScript: Check out
client.helpers.*
- .NET: Check out
BulkAllObservable
- PHP: Check out bulk indexing.
Submitting bulk requests with cURL
If you're providing text file input to curl
, you must use the --data-binary
flag instead of plain -d
.
The latter doesn't preserve newlines. For example:
$ cat requests
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
$ curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary "@requests"; echo
{"took":7, "errors": false, "items":[{"index":{"_index":"test","_id":"1","_version":1,"result":"created","forced_refresh":false}}]}
Optimistic concurrency control
Each index
and delete
action within a bulk API call may include the if_seq_no
and if_primary_term
parameters in their respective action and meta data lines.
The if_seq_no
and if_primary_term
parameters control how operations are run, based on the last modification to existing documents. See Optimistic concurrency control for more details.
Versioning
Each bulk item can include the version value using the version
field.
It automatically follows the behavior of the index or delete operation based on the _version
mapping.
It also support the version_type
.
Routing
Each bulk item can include the routing value using the routing
field.
It automatically follows the behavior of the index or delete operation based on the _routing
mapping.
NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing
setting enabled in the template.
Wait for active shards
When making bulk calls, you can set the wait_for_active_shards
parameter to require a minimum number of shard copies to be active before starting to process the bulk request.
Refresh
Control when the changes made by this request are visible to search.
NOTE: Only the shards that receive the bulk request will be affected by refresh.
Imagine a _bulk?refresh=wait_for
request with three documents in it that happen to be routed to different shards in an index with five shards.
The request will only wait for those three shards to refresh.
The other two shards that make up the index do not participate in the _bulk
request at all.
Query parameters
-
True or false if to include the document source in the error message in case of parsing errors.
-
If
true
, the response will include the ingest pipelines that were run for each index or create. -
The pipeline identifier to use to preprocess incoming documents. If the index has a default ingest pipeline specified, setting the value to
_none
turns off the default ingest pipeline for this request. If a final pipeline is configured, it will always run regardless of the value of this parameter. -
If
true
, Elasticsearch refreshes the affected shards to make this operation visible to search. Ifwait_for
, wait for a refresh to make this operation visible to search. Iffalse
, do nothing with refreshes. Valid values:true
,false
,wait_for
.Values are
true
,false
, orwait_for
. -
A custom value that is used to route operations to a specific shard.
-
Indicates whether to return the
_source
field (true
orfalse
) or contains a list of fields to return. -
A comma-separated list of source fields to exclude from the response. You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter. If the_source
parameter isfalse
, this parameter is ignored. -
A comma-separated list of source fields to include in the response. If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter. If the_source
parameter isfalse
, this parameter is ignored. -
The period each action waits for the following operations: automatic index creation, dynamic mapping updates, and waiting for active shards. The default is
1m
(one minute), which guarantees Elasticsearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur.Values are
-1
or0
. -
The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). The default is1
, which waits for each primary shard to be active.Values are
all
orindex-setting
. -
If
true
, the request's actions must target an index alias. -
If
true
, the request's actions must target a data stream (existing or to be created).
curl \
--request PUT 'http://api.example.com/{index}/_bulk' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{ \"index\" : { \"_index\" : \"test\", \"_id\" : \"1\" } }\n{ \"field1\" : \"value1\" }\n{ \"delete\" : { \"_index\" : \"test\", \"_id\" : \"2\" } }\n{ \"create\" : { \"_index\" : \"test\", \"_id\" : \"3\" } }\n{ \"field1\" : \"value3\" }\n{ \"update\" : {\"_id\" : \"1\", \"_index\" : \"test\"} }\n{ \"doc\" : {\"field2\" : \"value2\"} }"'
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }
{ "update" : {"_id" : "1", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"} }
{ "update" : { "_id" : "0", "_index" : "index1", "retry_on_conflict" : 3} }
{ "script" : { "source": "ctx._source.counter += params.param1", "lang" : "painless", "params" : {"param1" : 1}}, "upsert" : {"counter" : 1}}
{ "update" : {"_id" : "2", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"}, "doc_as_upsert" : true }
{ "update" : {"_id" : "3", "_index" : "index1", "_source" : true} }
{ "doc" : {"field" : "value"} }
{ "update" : {"_id" : "4", "_index" : "index1"} }
{ "doc" : {"field" : "value"}, "_source": true}
{ "update": {"_id": "5", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "update": {"_id": "6", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "create": {"_id": "7", "_index": "index1"} }
{ "my_field": "foo" }
{ "index" : { "_index" : "my_index", "_id" : "1", "dynamic_templates": {"work_location": "geo_point"}} }
{ "field" : "value1", "work_location": "41.12,-71.34", "raw_location": "41.12,-71.34"}
{ "create" : { "_index" : "my_index", "_id" : "2", "dynamic_templates": {"home_location": "geo_point"}} }
{ "field" : "value2", "home_location": "41.12,-71.34"}
{
"took": 30,
"errors": false,
"items": [
{
"index": {
"_index": "test",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 0,
"_primary_term": 1
}
},
{
"delete": {
"_index": "test",
"_id": "2",
"_version": 1,
"result": "not_found",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 404,
"_seq_no" : 1,
"_primary_term" : 2
}
},
{
"create": {
"_index": "test",
"_id": "3",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 2,
"_primary_term" : 3
}
},
{
"update": {
"_index": "test",
"_id": "1",
"_version": 2,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 200,
"_seq_no" : 3,
"_primary_term" : 4
}
}
]
}
{
"took": 486,
"errors": true,
"items": [
{
"update": {
"_index": "index1",
"_id": "5",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"_index": "index1",
"_id": "6",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"create": {
"_index": "index1",
"_id": "7",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1,
"status": 201
}
}
]
}
{
"items": [
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
}
]
}
Bulk index or delete documents
Generally available
Perform multiple index
, create
, delete
, and update
actions in a single request.
This reduces overhead and can greatly increase indexing speed.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:
- To use the
create
action, you must have thecreate_doc
,create
,index
, orwrite
index privilege. Data streams support only thecreate
action. - To use the
index
action, you must have thecreate
,index
, orwrite
index privilege. - To use the
delete
action, you must have thedelete
orwrite
index privilege. - To use the
update
action, you must have theindex
orwrite
index privilege. - To automatically create a data stream or index with a bulk API request, you must have the
auto_configure
,create_index
, ormanage
index privilege. - To make the result of a bulk operation visible to search using the
refresh
parameter, you must have themaintenance
ormanage
index privilege.
Automatic data stream creation requires a matching index template with data stream enabled.
The actions are specified in the request body using a newline delimited JSON (NDJSON) structure:
action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n
The index
and create
actions expect a source on the next line and have the same semantics as the op_type
parameter in the standard index API.
A create
action fails if a document with the same ID already exists in the target
An index
action adds or replaces a document as necessary.
NOTE: Data streams support only the create
action.
To update or delete a document in a data stream, you must target the backing index containing the document.
An update
action expects that the partial doc, upsert, and script and its options are specified on the next line.
A delete
action does not expect a source on the next line and has the same semantics as the standard delete API.
NOTE: The final line of data must end with a newline character (\n
).
Each newline character may be preceded by a carriage return (\r
).
When sending NDJSON data to the _bulk
endpoint, use a Content-Type
header of application/json
or application/x-ndjson
.
Because this format uses literal newline characters (\n
) as delimiters, make sure that the JSON actions and sources are not pretty printed.
If you provide a target in the request path, it is used for any actions that don't explicitly specify an _index
argument.
A note on the format: the idea here is to make processing as fast as possible.
As some of the actions are redirected to other shards on other nodes, only action_meta_data
is parsed on the receiving node side.
Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible.
There is no "correct" number of actions to perform in a single bulk request. Experiment with different settings to find the optimal size for your particular workload. Note that Elasticsearch limits the maximum size of a HTTP request to 100mb by default so clients must ensure that no request exceeds this size. It is not possible to index a single document that exceeds the size limit, so you must pre-process any such documents into smaller pieces before sending them to Elasticsearch. For instance, split documents into pages or chapters before indexing them, or store raw binary data in a system outside Elasticsearch and replace the raw data with a link to the external system in the documents that you send to Elasticsearch.
Client suppport for bulk requests
Some of the officially supported clients provide helpers to assist with bulk requests and reindexing:
- Go: Check out
esutil.BulkIndexer
- Perl: Check out
Search::Elasticsearch::Client::5_0::Bulk
andSearch::Elasticsearch::Client::5_0::Scroll
- Python: Check out
elasticsearch.helpers.*
- JavaScript: Check out
client.helpers.*
- .NET: Check out
BulkAllObservable
- PHP: Check out bulk indexing.
Submitting bulk requests with cURL
If you're providing text file input to curl
, you must use the --data-binary
flag instead of plain -d
.
The latter doesn't preserve newlines. For example:
$ cat requests
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
$ curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary "@requests"; echo
{"took":7, "errors": false, "items":[{"index":{"_index":"test","_id":"1","_version":1,"result":"created","forced_refresh":false}}]}
Optimistic concurrency control
Each index
and delete
action within a bulk API call may include the if_seq_no
and if_primary_term
parameters in their respective action and meta data lines.
The if_seq_no
and if_primary_term
parameters control how operations are run, based on the last modification to existing documents. See Optimistic concurrency control for more details.
Versioning
Each bulk item can include the version value using the version
field.
It automatically follows the behavior of the index or delete operation based on the _version
mapping.
It also support the version_type
.
Routing
Each bulk item can include the routing value using the routing
field.
It automatically follows the behavior of the index or delete operation based on the _routing
mapping.
NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing
setting enabled in the template.
Wait for active shards
When making bulk calls, you can set the wait_for_active_shards
parameter to require a minimum number of shard copies to be active before starting to process the bulk request.
Refresh
Control when the changes made by this request are visible to search.
NOTE: Only the shards that receive the bulk request will be affected by refresh.
Imagine a _bulk?refresh=wait_for
request with three documents in it that happen to be routed to different shards in an index with five shards.
The request will only wait for those three shards to refresh.
The other two shards that make up the index do not participate in the _bulk
request at all.
Query parameters
-
True or false if to include the document source in the error message in case of parsing errors.
-
If
true
, the response will include the ingest pipelines that were run for each index or create. -
The pipeline identifier to use to preprocess incoming documents. If the index has a default ingest pipeline specified, setting the value to
_none
turns off the default ingest pipeline for this request. If a final pipeline is configured, it will always run regardless of the value of this parameter. -
If
true
, Elasticsearch refreshes the affected shards to make this operation visible to search. Ifwait_for
, wait for a refresh to make this operation visible to search. Iffalse
, do nothing with refreshes. Valid values:true
,false
,wait_for
.Values are
true
,false
, orwait_for
. -
A custom value that is used to route operations to a specific shard.
-
Indicates whether to return the
_source
field (true
orfalse
) or contains a list of fields to return. -
A comma-separated list of source fields to exclude from the response. You can also use this parameter to exclude fields from the subset specified in
_source_includes
query parameter. If the_source
parameter isfalse
, this parameter is ignored. -
A comma-separated list of source fields to include in the response. If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludes
query parameter. If the_source
parameter isfalse
, this parameter is ignored. -
The period each action waits for the following operations: automatic index creation, dynamic mapping updates, and waiting for active shards. The default is
1m
(one minute), which guarantees Elasticsearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur.Values are
-1
or0
. -
The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
). The default is1
, which waits for each primary shard to be active.Values are
all
orindex-setting
. -
If
true
, the request's actions must target an index alias. -
If
true
, the request's actions must target a data stream (existing or to be created).
curl \
--request POST 'http://api.example.com/{index}/_bulk' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{ \"index\" : { \"_index\" : \"test\", \"_id\" : \"1\" } }\n{ \"field1\" : \"value1\" }\n{ \"delete\" : { \"_index\" : \"test\", \"_id\" : \"2\" } }\n{ \"create\" : { \"_index\" : \"test\", \"_id\" : \"3\" } }\n{ \"field1\" : \"value3\" }\n{ \"update\" : {\"_id\" : \"1\", \"_index\" : \"test\"} }\n{ \"doc\" : {\"field2\" : \"value2\"} }"'
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }
{ "update" : {"_id" : "1", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"} }
{ "update" : { "_id" : "0", "_index" : "index1", "retry_on_conflict" : 3} }
{ "script" : { "source": "ctx._source.counter += params.param1", "lang" : "painless", "params" : {"param1" : 1}}, "upsert" : {"counter" : 1}}
{ "update" : {"_id" : "2", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"}, "doc_as_upsert" : true }
{ "update" : {"_id" : "3", "_index" : "index1", "_source" : true} }
{ "doc" : {"field" : "value"} }
{ "update" : {"_id" : "4", "_index" : "index1"} }
{ "doc" : {"field" : "value"}, "_source": true}
{ "update": {"_id": "5", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "update": {"_id": "6", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "create": {"_id": "7", "_index": "index1"} }
{ "my_field": "foo" }
{ "index" : { "_index" : "my_index", "_id" : "1", "dynamic_templates": {"work_location": "geo_point"}} }
{ "field" : "value1", "work_location": "41.12,-71.34", "raw_location": "41.12,-71.34"}
{ "create" : { "_index" : "my_index", "_id" : "2", "dynamic_templates": {"home_location": "geo_point"}} }
{ "field" : "value2", "home_location": "41.12,-71.34"}
{
"took": 30,
"errors": false,
"items": [
{
"index": {
"_index": "test",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 0,
"_primary_term": 1
}
},
{
"delete": {
"_index": "test",
"_id": "2",
"_version": 1,
"result": "not_found",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 404,
"_seq_no" : 1,
"_primary_term" : 2
}
},
{
"create": {
"_index": "test",
"_id": "3",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 2,
"_primary_term" : 3
}
},
{
"update": {
"_index": "test",
"_id": "1",
"_version": 2,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 200,
"_seq_no" : 3,
"_primary_term" : 4
}
}
]
}
{
"took": 486,
"errors": true,
"items": [
{
"update": {
"_index": "index1",
"_id": "5",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"_index": "index1",
"_id": "6",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"create": {
"_index": "index1",
"_id": "7",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1,
"status": 201
}
}
]
}
{
"items": [
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
}
]
}
Delete an async EQL search
Generally available; Added in 7.9.0
Delete an async EQL search or a stored synchronous EQL search. The API also deletes results for the search.
curl \
--request DELETE 'http://api.example.com/_eql/search/{id}' \
--header "Authorization: $API_KEY"
curl \
--request GET 'http://api.example.com/_eql/search/status/{id}' \
--header "Authorization: $API_KEY"
{
"id": "FmNJRUZ1YWZCU3dHY1BIOUhaenVSRkEaaXFlZ3h4c1RTWFNocDdnY2FSaERnUTozNDE=",
"is_running" : true,
"is_partial" : true,
"start_time_in_millis" : 1611690235000,
"expiration_time_in_millis" : 1611690295000
}
Get global checkpoints
Generally available; Added in 7.13.0
Get the current global checkpoints for an index. This API is designed for internal use by the Fleet server project.
Query parameters
-
A boolean value which controls whether to wait (until the timeout) for the global checkpoints to advance past the provided
checkpoints
. -
A boolean value which controls whether to wait (until the timeout) for the target index to exist and all primary shards be active. Can only be true when
wait_for_advance
is true. -
A comma separated list of previous global checkpoints. When used in combination with
wait_for_advance
, the API will only return once the global checkpoints advances past the checkpoints. Providing an empty list will cause Elasticsearch to immediately return the current global checkpoints. -
Period to wait for a global checkpoints to advance past
checkpoints
.Values are
-1
or0
.
curl \
--request GET 'http://api.example.com/{index}/_fleet/global_checkpoints' \
--header "Authorization: $API_KEY"
Explore graph analytics
Generally available
Extract and summarize information about the documents and terms in an Elasticsearch data stream or index.
The easiest way to understand the behavior of this API is to use the Graph UI to explore connections.
An initial request to the _explore
API contains a seed query that identifies the documents of interest and specifies the fields that define the vertices and connections you want to include in the graph.
Subsequent requests enable you to spider out from one more vertices of interest.
You can exclude vertices that have already been returned.
Query parameters
-
Custom value used to route operations to a specific shard.
-
Specifies the period of time to wait for a response from each shard. If no response is received before the timeout expires, the request fails and returns an error. Defaults to no timeout.
Values are
-1
or0
.
Body
-
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
External documentation -
Specifies one or more fields that contain the terms you want to include in the graph as vertices.
curl \
--request GET 'http://api.example.com/{index}/_graph/explore' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"query\": {\n \"match\": {\n \"query.raw\": \"midi\"\n }\n },\n \"vertices\": [\n {\n \"field\": \"product\"\n }\n ],\n \"connections\": {\n \"vertices\": [\n {\n \"field\": \"query.raw\"\n }\n ]\n }\n}"'
{
"query": {
"match": {
"query.raw": "midi"
}
},
"vertices": [
{
"field": "product"
}
],
"connections": {
"vertices": [
{
"field": "query.raw"
}
]
}
}
Get the dangling indices
Generally available; Added in 7.9.0
If Elasticsearch encounters index data that is absent from the current cluster state, those indices are considered to be dangling.
For example, this can happen if you delete more than cluster.indices.tombstones.size
indices while an Elasticsearch node is offline.
Use this API to list dangling indices, which you can then import or delete.
Required authorization
- Cluster privileges:
manage
curl \
--request GET 'http://api.example.com/_dangling' \
--header "Authorization: $API_KEY"
{
"dangling_indices": [
{
"index_name": "my-index-000001",
"index_uuid": "zmM4e0JtBkeUjiHD-MihPQ",
"creation_date_millis": 1589414451372,
"node_ids": [
"pL47UN3dAb2d5RCWP6lQ3e"
]
}
]
}
Create an index
Generally available
You can use the create index API to add a new index to an Elasticsearch cluster. When creating an index, you can specify the following:
- Settings for the index.
- Mappings for fields in the index.
- Index aliases
Wait for active shards
By default, index creation will only return a response to the client when the primary copies of each shard have been started, or the request times out.
The index creation response will indicate what happened.
For example, acknowledged
indicates whether the index was successfully created in the cluster, while shards_acknowledged
indicates whether the requisite number of shard copies were started for each shard in the index before timing out.
Note that it is still possible for either acknowledged
or shards_acknowledged
to be false
, but for the index creation to be successful.
These values simply indicate whether the operation completed before the timeout.
If acknowledged
is false, the request timed out before the cluster state was updated with the newly created index, but it probably will be created sometime soon.
If shards_acknowledged
is false, then the request timed out before the requisite number of shards were started (by default just the primaries), even if the cluster state was successfully updated to reflect the newly created index (that is to say, acknowledged
is true
).
You can change the default of only waiting for the primary shards to start through the index setting index.write.wait_for_active_shards
.
Note that changing this setting will also affect the wait_for_active_shards
value on all subsequent write operations.
Required authorization
- Index privileges:
create_index
,manage
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
).Values are
all
orindex-setting
.
Body
-
Aliases for the index.
-
Index settings
curl \
--request PUT 'http://api.example.com/{index}' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"settings\": {\n \"number_of_shards\": 3,\n \"number_of_replicas\": 2\n }\n}"'
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}
{
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"field1": { "type": "text" }
}
}
}
{
"aliases": {
"alias_1": {},
"alias_2": {
"filter": {
"term": {
"user.id": "kimchy"
}
},
"routing": "shard-1"
}
}
}
Check indices
Generally available
Check if one or more indices, index aliases, or data streams exist.
Query parameters
-
If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden
.Values are
all
,open
,closed
,hidden
, ornone
. -
If
true
, returns settings in flat format. -
If
true
, return all default settings in the response. -
If
true
, the request retrieves information from the local node only.
curl \
--request HEAD 'http://api.example.com/{index}' \
--header "Authorization: $API_KEY"
Path parameters
-
Comma-separated list of data streams or indices used to limit the request. Supports wildcards (
*
). To target all data streams and indices, omit this parameter or use*
or_all
. -
Comma-separated list of aliases to retrieve. Supports wildcards (
*
). To retrieve all aliases, omit this parameter or use*
or_all
.
Query parameters
-
If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden
.Values are
all
,open
,closed
,hidden
, ornone
. -
If
true
, the request retrieves information from the local node only.
curl \
--request GET 'http://api.example.com/{index}/_alias/{name}' \
--header "Authorization: $API_KEY"
Create or update an alias
Generally available
Adds a data stream or index to an alias.
Path parameters
-
Comma-separated list of data streams or indices to add. Supports wildcards (
*
). Wildcard patterns that match both data streams and indices return an error. -
Alias to update. If the alias doesn’t exist, the request creates it. Index alias names support date math.
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
.
Body
-
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
External documentation -
If
true
, sets the write index or data stream for the alias. If an alias points to multiple indices or data streams andis_write_index
isn’t set, the alias rejects write requests. If an index alias points to one index andis_write_index
isn’t set, the index automatically acts as the write index. Data stream aliases don’t automatically set a write data stream, even if the alias points to one data stream.
curl \
--request PUT 'http://api.example.com/{index}/_alias/{name}' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"my-data-stream\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}"'
{
"actions": [
{
"add": {
"index": "my-data-stream",
"alias": "my-alias"
}
}
]
}
Create or update an alias
Generally available
Adds a data stream or index to an alias.
Path parameters
-
Comma-separated list of data streams or indices to add. Supports wildcards (
*
). Wildcard patterns that match both data streams and indices return an error. -
Alias to update. If the alias doesn’t exist, the request creates it. Index alias names support date math.
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
.
Body
-
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
External documentation -
If
true
, sets the write index or data stream for the alias. If an alias points to multiple indices or data streams andis_write_index
isn’t set, the alias rejects write requests. If an index alias points to one index andis_write_index
isn’t set, the index automatically acts as the write index. Data stream aliases don’t automatically set a write data stream, even if the alias points to one data stream.
curl \
--request PUT 'http://api.example.com/{index}/_aliases/{name}' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"actions\": [\n {\n \"add\": {\n \"index\": \"my-data-stream\",\n \"alias\": \"my-alias\"\n }\n }\n ]\n}"'
{
"actions": [
{
"add": {
"index": "my-data-stream",
"alias": "my-alias"
}
}
]
}
Get mapping definitions
Generally available
Retrieves mapping definitions for one or more fields. For data streams, the API retrieves field mappings for the stream’s backing indices.
This API is useful if you don't need a complete mapping or if an index mapping contains a large number of fields.
Required authorization
- Index privileges:
view_index_metadata
Path parameters
-
Comma-separated list of data streams, indices, and aliases used to limit the request. Supports wildcards (
*
). To target all data streams and indices, omit this parameter or use*
or_all
. -
Comma-separated list or wildcard expression of fields used to limit returned information. Supports wildcards (
*
).
Query parameters
-
If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden
.Values are
all
,open
,closed
,hidden
, ornone
. -
If
true
, return all default settings in the response. -
If
true
, the request retrieves information from the local node only.
curl \
--request GET 'http://api.example.com/{index}/_mapping/field/{fields}' \
--header "Authorization: $API_KEY"
{
"publications": {
"mappings": {
"title": {
"full_name": "title",
"mapping": {
"title": {
"type": "text"
}
}
}
}
}
}
{
"publications": {
"mappings": {
"author.id": {
"full_name": "author.id",
"mapping": {
"id": {
"type": "text"
}
}
},
"abstract": {
"full_name": "abstract",
"mapping": {
"abstract": {
"type": "text"
}
}
}
}
}
}
{
"publications": {
"mappings": {
"author.name": {
"full_name": "author.name",
"mapping": {
"name": {
"type": "text"
}
}
},
"abstract": {
"full_name": "abstract",
"mapping": {
"abstract": {
"type": "text"
}
}
},
"author.id": {
"full_name": "author.id",
"mapping": {
"id": {
"type": "text"
}
}
}
}
}
}
Update field mappings
Generally available
Add new fields to an existing data stream or index. You can also use this API to change the search settings of existing fields and add new properties to existing object fields. For data streams, these changes are applied to all backing indices by default.
Add multi-fields to an existing field
Multi-fields let you index the same field in different ways. You can use this API to update the fields mapping parameter and enable multi-fields for an existing field. WARNING: If an index (or data stream) contains documents when you add a multi-field, those documents will not have values for the new multi-field. You can populate the new multi-field with the update by query API.
Change supported mapping parameters for an existing field
The documentation for each mapping parameter indicates whether you can update it for an existing field using this API.
For example, you can use the update mapping API to update the ignore_above
parameter.
Change the mapping of an existing field
Except for supported mapping parameters, you can't change the mapping or field type of an existing field. Changing an existing field could invalidate data that's already indexed.
If you need to change the mapping of a field in a data stream's backing indices, refer to documentation about modifying data streams. If you need to change the mapping of a field in other indices, create a new index with the correct mapping and reindex your data into that index.
Rename a field
Renaming a field would invalidate data already indexed under the old field name. Instead, add an alias field to create an alternate field name.
Required authorization
- Index privileges:
manage
Path parameters
-
A comma-separated list of index names the mapping should be added to (supports wildcards); use
_all
or omit to add the mapping on all indices.
Query parameters
-
If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden
.Values are
all
,open
,closed
,hidden
, ornone
. -
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
If
true
, the mappings are applied only to the current write index for the target.
Body
Required
-
Controls whether dynamic date detection is enabled.
-
Values are
strict
,runtime
,true
, orfalse
. -
If date detection is enabled then new string fields are checked against 'dynamic_date_formats' and if the value matches then a new date field is added instead of string.
-
Specify dynamic templates for the mapping.
-
Automatically map strings into numeric data types for all fields.
-
Mapping for a field. For new fields, this mapping can include:
- Field name
- Field data type
- Mapping parameters
curl \
--request PUT 'http://api.example.com/{index}/_mapping' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"properties\": {\n \"user\": {\n \"properties\": {\n \"name\": {\n \"type\": \"keyword\"\n }\n }\n }\n }\n}"'
{
"properties": {
"user": {
"properties": {
"name": {
"type": "keyword"
}
}
}
}
}
Get legacy index templates
Deprecated
Generally available
Get information about one or more index templates.
IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.
Required authorization
- Cluster privileges:
manage_index_templates
curl \
--request GET 'http://api.example.com/_template' \
--header "Authorization: $API_KEY"
Resolve the cluster
Generally available; Added in 8.13.0
Resolve the specified index expressions to return information about each cluster, including the local "querying" cluster, if included. If no index expression is provided, the API will return information about all the remote clusters that are configured on the querying cluster.
This endpoint is useful before doing a cross-cluster search in order to determine which remote clusters should be included in a search.
You use the same index expression with this endpoint as you would for cross-cluster search. Index and cluster exclusions are also supported with this endpoint.
For each cluster in the index expression, information is returned about:
- Whether the querying ("local") cluster is currently connected to each remote cluster specified in the index expression. Note that this endpoint actively attempts to contact the remote clusters, unlike the
remote/info
endpoint. - Whether each remote cluster is configured with
skip_unavailable
astrue
orfalse
. - Whether there are any indices, aliases, or data streams on that cluster that match the index expression.
- Whether the search is likely to have errors returned when you do the cross-cluster search (including any authorization errors if you do not have permission to query the index).
- Cluster version information, including the Elasticsearch server version.
For example, GET /_resolve/cluster/my-index-*,cluster*:my-index-*
returns information about the local cluster and all remotely configured clusters that start with the alias cluster*
.
Each cluster returns information about whether it has any indices, aliases or data streams that match my-index-*
.
Note on backwards compatibility
The ability to query without an index expression was added in version 8.18, so when
querying remote clusters older than that, the local cluster will send the index
expression dummy*
to those remote clusters. Thus, if an errors occur, you may see a reference
to that index expression even though you didn't request it. If it causes a problem, you can
instead include an index expression like *:*
to bypass the issue.
Advantages of using this endpoint before a cross-cluster search
You may want to exclude a cluster or index from a search when:
- A remote cluster is not currently connected and is configured with
skip_unavailable=false
. Running a cross-cluster search under those conditions will cause the entire search to fail. - A cluster has no matching indices, aliases or data streams for the index expression (or your user does not have permissions to search them). For example, suppose your index expression is
logs*,remote1:logs*
and the remote1 cluster has no indices, aliases or data streams that matchlogs*
. In that case, that cluster will return no results from that cluster if you include it in a cross-cluster search. - The index expression (combined with any query parameters you specify) will likely cause an exception to be thrown when you do the search. In these cases, the "error" field in the
_resolve/cluster
response will be present. (This is also where security/permission errors will be shown.) - A remote cluster is an older version that does not support the feature you want to use in your search.
Test availability of remote clusters
The remote/info
endpoint is commonly used to test whether the "local" cluster (the cluster being queried) is connected to its remote clusters, but it does not necessarily reflect whether the remote cluster is available or not.
The remote cluster may be available, while the local cluster is not currently connected to it.
You can use the _resolve/cluster
API to attempt to reconnect to remote clusters.
For example with GET _resolve/cluster
or GET _resolve/cluster/*:*
.
The connected
field in the response will indicate whether it was successful.
If a connection was (re-)established, this will also cause the remote/info
endpoint to now indicate a connected status.
Required authorization
- Index privileges:
view_index_metadata
Path parameters
-
A comma-separated list of names or index patterns for the indices, aliases, and data streams to resolve. Resources on remote clusters can be specified using the
<cluster>
:<name>
syntax. Index and cluster exclusions (e.g.,-cluster1:*
) are also supported. If no index expression is specified, information about all remote clusters configured on the local cluster is returned without doing any index matching
Query parameters
-
If false, the request returns an error if any wildcard expression, index alias, or
_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*
returns an error if an index starts withfoo
but no index starts withbar
. NOTE: This option is only supported when specifying an index expression. You will get an error if you specify index options to the_resolve/cluster
API endpoint that takes no index expression. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden
. NOTE: This option is only supported when specifying an index expression. You will get an error if you specify index options to the_resolve/cluster
API endpoint that takes no index expression.Values are
all
,open
,closed
,hidden
, ornone
. -
If true, concrete, expanded, or aliased indices are ignored when frozen. NOTE: This option is only supported when specifying an index expression. You will get an error if you specify index options to the
_resolve/cluster
API endpoint that takes no index expression. -
The maximum time to wait for remote clusters to respond. If a remote cluster does not respond within this timeout period, the API response will show the cluster as not connected and include an error message that the request timed out.
The default timeout is unset and the query can take as long as the networking layer is configured to wait for remote clusters that are not responding (typically 30 seconds).
Values are
-1
or0
.
curl \
--request GET 'http://api.example.com/_resolve/cluster/{name}' \
--header "Authorization: $API_KEY"
{
"(local)": {
"connected": true,
"skip_unavailable": false,
"matching_indices": true,
"version": {
"number": "8.13.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
},
"cluster_one": {
"connected": true,
"skip_unavailable": true,
"matching_indices": true,
"version": {
"number": "8.13.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
},
"cluster_two": {
"connected": true,
"skip_unavailable": false,
"matching_indices": true,
"version": {
"number": "8.13.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
}
}
{
"(local)": {
"connected": true,
"skip_unavailable": false,
"error": "no such index [not_present]"
},
"cluster_one": {
"connected": true,
"skip_unavailable": true,
"matching_indices": false,
"version": {
"number": "8.13.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
},
"cluster_two": {
"connected": false,
"skip_unavailable": false
},
"cluster_three": {
"connected": false,
"skip_unavailable": false,
"error": "Request timed out before receiving a response from the remote cluster"
},
"oldcluster": {
"connected": true,
"skip_unavailable": false,
"matching_indices": true
}
}
Query parameters
-
If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden
.Values are
all
,open
,closed
,hidden
, ornone
. -
If
true
, the request returns a verbose response.
curl \
--request GET 'http://api.example.com/_segments' \
--header "Authorization: $API_KEY"
{
"acknowledged": true,
"shards_acknowledged": true,
"old_index": ".ds-my-data-stream-2099.05.06-000001",
"new_index": ".ds-my-data-stream-2099.05.07-000002",
"rolled_over": true,
"dry_run": false,
"lazy": false,
"conditions": {
"[max_age: 7d]": false,
"[max_docs: 1000]": true,
"[max_primary_shard_size: 50gb]": false,
"[max_primary_shard_docs: 2000]": false
}
}
Shrink an index
Generally available; Added in 5.0.0
Shrink an index into a new index with fewer primary shards.
Before you can shrink an index:
- The index must be read-only.
- A copy of every shard in the index must reside on the same node.
- The index must have a green health status.
To make shard allocation easier, we recommend you also remove the index's replica shards. You can later re-add replica shards as part of the shrink operation.
The requested number of primary shards in the target index must be a factor of the number of shards in the source index. For example an index with 8 primary shards can be shrunk into 4, 2 or 1 primary shards or an index with 15 primary shards can be shrunk into 5, 3 or 1. If the number of shards in the index is a prime number it can only be shrunk into a single primary shard Before shrinking, a (primary or replica) copy of every shard in the index must be present on the same node.
The current write index on a data stream cannot be shrunk. In order to shrink the current write index, the data stream must first be rolled over so that a new write index is created and then the previous write index can be shrunk.
A shrink operation:
- Creates a new target index with the same definition as the source index, but with a smaller number of primary shards.
- Hard-links segments from the source index into the target index. If the file system does not support hard-linking, then all segments are copied into the new index, which is a much more time consuming process. Also if using multiple data paths, shards on different data paths require a full copy of segment files if they are not on the same disk since hardlinks do not work across disks.
- Recovers the target index as though it were a closed index which had just been re-opened. Recovers shards to the
.routing.allocation.initial_recovery._id
index setting.
IMPORTANT: Indices can only be shrunk if they satisfy the following requirements:
- The target index must not exist.
- The source index must have more primary shards than the target index.
- The number of primary shards in the target index must be a factor of the number of primary shards in the source index. The source index must have more primary shards than the target index.
- The index must not contain more than 2,147,483,519 documents in total across all shards that will be shrunk into a single shard on the target index as this is the maximum number of docs that can fit into a single shard.
- The node handling the shrink process must have sufficient free disk space to accommodate a second copy of the existing index.
Required authorization
- Index privileges:
manage
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1
or0
. -
The number of shard copies that must be active before proceeding with the operation. Set to
all
or any positive integer up to the total number of shards in the index (number_of_replicas+1
).Values are
all
orindex-setting
.
curl \
--request POST 'http://api.example.com/{index}/_shrink/{target}' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '"{\n \"settings\": {\n \"index.routing.allocation.require._name\": null,\n \"index.blocks.write\": null\n }\n}"'
{
"settings": {
"index.routing.allocation.require._name": null,
"index.blocks.write": null
}
}
Get index statistics
Generally available; Added in 1.3.0
For data streams, the API retrieves statistics for the stream's backing indices.
By default, the returned statistics are index-level with primaries
and total
aggregations.
primaries
are the values for only the primary shards.
total
are the accumulated values for both primary and replica shards.
To get shard-level statistics, set the level
parameter to shards
.
NOTE: When moving to another node, the shard-level statistics for a shard are cleared. Although the shard is no longer part of the node, that node retains any node-level statistics to which the shard contributed.
Required authorization
- Index privileges:
monitor
Path parameters
-
A comma-separated list of index names; use
_all
or empty string to perform the operation on all indices
Query parameters
-
Comma-separated list or wildcard expressions of fields to include in fielddata and suggest statistics.
-
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden
.Values are
all
,open
,closed
,hidden
, ornone
. -
Comma-separated list or wildcard expressions of fields to include in fielddata statistics.
-
Comma-separated list or wildcard expressions of fields to include in the statistics.
-
If true, statistics are not collected from closed indices.
-
Comma-separated list of search groups to include in the search statistics.
-
If true, the call reports the aggregated disk usage of each one of the Lucene index files (only applies if segment stats are requested).
-
If true, the response includes information from segments that are not loaded into memory.
-
Indicates whether statistics are aggregated at the cluster, index, or shard level.
Values are
cluster
,indices
, orshards
.
curl \
--request GET 'http://api.example.com/{index}/_stats' \
--header "Authorization: $API_KEY"
Validate a query
Generally available; Added in 1.3.0
Validates a query without running it.
Query parameters
-
If
false
, the request returns an error if any wildcard expression, index alias, or_all
value targets only missing or closed indices. This behavior applies even if the request targets other open indices. -
If
true
, the validation is executed on all shards instead of one random shard per index. -
Analyzer to use for the query string. This parameter can only be used when the
q
query string parameter is specified. -
If
true
, wildcard and prefix queries are analyzed. -
The default operator for query string query:
AND
orOR
.Values are
and
,AND
,or
, orOR
. -
Field to use as default where no field prefix is given in the query string. This parameter can only be used when the
q
query string parameter is specified. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden
.Values are
all
,open
,closed
,hidden
, ornone
. -
If
true
, the response returns detailed information if an error has occurred. -
If
true
, format-based query failures (such as providing text to a numeric field) in the query string will be ignored. -
If
true
, returns a more detailed explanation showing the actual Lucene query that will be executed. -
Query in the Lucene query string syntax.
Body
-
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
External documentation
curl \
--request GET 'http://api.example.com/_validate/query' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '{"query":{}}'
Inference
Inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.
curl \
--request GET 'http://api.example.com/_inference/{inference_id}' \
--header "Authorization: $API_KEY"