http://api.example.comElasticsearch provides REST APIs that are used by the UI components and can be called directly to configure and access Elasticsearch features.
This documentation is derived from the 8.x branch of the elasticsearch-specification repository. It is provided under license Attribution-NonCommercial-NoDerivatives 4.0 International.
This documentation contains work-in-progress information for future Elastic Stack releases.
Last update on Aug 28, 2025.
This API is provided under license Apache 2.0.
The API accepts 3 different authentication methods:
Elasticsearch APIs support key-based authentication. You must create an API key and use the encoded value in the request header. For example:
curl -X GET "${ES_URL}/_cat/indices?v=true" \
-H "Authorization: ApiKey ${API_KEY}"
To get API keys, use the /_security/api_key APIs.
Elasticsearch APIs support the use of bearer tokens in the Authorization HTTP header to authenticate with the API.
For examples, refer to Token-based authentication services
NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.
GET /_autoscaling/policy/my_autoscaling_policy
resp = client.autoscaling.get_autoscaling_policy(
name="my_autoscaling_policy",
)const response = await client.autoscaling.getAutoscalingPolicy({
name: "my_autoscaling_policy",
});response = client.autoscaling.get_autoscaling_policy(
name: "my_autoscaling_policy"
)$resp = $client->autoscaling()->getAutoscalingPolicy([
"name" => "my_autoscaling_policy",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_autoscaling/policy/my_autoscaling_policy"client.autoscaling().getAutoscalingPolicy(g -> g
.name("my_autoscaling_policy")
);
{
"roles": <roles>,
"deciders": <deciders>
}NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Decider settings.
PUT /_autoscaling/policy/<name>
{
"roles": [],
"deciders": {
"fixed": {
}
}
}resp = client.autoscaling.put_autoscaling_policy(
name="<name>",
policy={
"roles": [],
"deciders": {
"fixed": {}
}
},
)const response = await client.autoscaling.putAutoscalingPolicy({
name: "<name>",
policy: {
roles: [],
deciders: {
fixed: {},
},
},
});response = client.autoscaling.put_autoscaling_policy(
name: "<name>",
body: {
"roles": [],
"deciders": {
"fixed": {}
}
}
)$resp = $client->autoscaling()->putAutoscalingPolicy([
"name" => "<name>",
"body" => [
"roles" => array(
),
"deciders" => [
"fixed" => new ArrayObject([]),
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"roles":[],"deciders":{"fixed":{}}}' "$ELASTICSEARCH_URL/_autoscaling/policy/<name>"client.autoscaling().putAutoscalingPolicy(p -> p
.name("<name>")
.policy(po -> po
.deciders("fixed", JsonData.fromJson("{}"))
)
);
{
"roles": [],
"deciders": {
"fixed": {
}
}
}{
"roles" : [ "data_hot" ],
"deciders": {
"fixed": {
}
}
}{
"acknowledged": true
}NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.
This API gets the current autoscaling capacity based on the configured autoscaling policy. It will return information to size the cluster appropriately to the current workload.
The required_capacity is calculated as the maximum of the required_capacity result of all individual deciders that are enabled for the policy.
The operator should verify that the current_nodes match the operator’s knowledge of the cluster to avoid making autoscaling decisions based on stale or incomplete information.
The response contains decider-specific information you can use to diagnose how and why autoscaling determined a certain capacity was required. This information is provided for diagnosis only. Do not use this information to make autoscaling decisions.
GET /_autoscaling/capacity
resp = client.autoscaling.get_autoscaling_capacity()const response = await client.autoscaling.getAutoscalingCapacity();response = client.autoscaling.get_autoscaling_capacity$resp = $client->autoscaling()->getAutoscalingCapacity();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_autoscaling/capacity"client.autoscaling().getAutoscalingCapacity(g -> g);
{
policies: {}
}All methods and paths for this operation:
GET _application/analytics/my*
resp = client.search_application.get_behavioral_analytics(
name="my*",
)const response = await client.searchApplication.getBehavioralAnalytics({
name: "my*",
});response = client.search_application.get_behavioral_analytics(
name: "my*"
)$resp = $client->searchApplication()->getBehavioralAnalytics([
"name" => "my*",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_application/analytics/my*"client.searchApplication().getBehavioralAnalytics(g -> g
.name("my*")
);
{
"my_analytics_collection": {
"event_data_stream": {
"name": "behavioral_analytics-events-my_analytics_collection"
}
},
"my_analytics_collection2": {
"event_data_stream": {
"name": "behavioral_analytics-events-my_analytics_collection2"
}
}
}PUT _application/analytics/my_analytics_collection
resp = client.search_application.put_behavioral_analytics(
name="my_analytics_collection",
)const response = await client.searchApplication.putBehavioralAnalytics({
name: "my_analytics_collection",
});response = client.search_application.put_behavioral_analytics(
name: "my_analytics_collection"
)$resp = $client->searchApplication()->putBehavioralAnalytics([
"name" => "my_analytics_collection",
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_application/analytics/my_analytics_collection"client.searchApplication().putBehavioralAnalytics(p -> p
.name("my_analytics_collection")
);
DELETE _application/analytics/my_analytics_collection/
resp = client.search_application.delete_behavioral_analytics(
name="my_analytics_collection",
)const response = await client.searchApplication.deleteBehavioralAnalytics({
name: "my_analytics_collection",
});response = client.search_application.delete_behavioral_analytics(
name: "my_analytics_collection"
)$resp = $client->searchApplication()->deleteBehavioralAnalytics([
"name" => "my_analytics_collection",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_application/analytics/my_analytics_collection/"client.searchApplication().deleteBehavioralAnalytics(d -> d
.name("my_analytics_collection")
);
The name of the behavioral analytics collection.
The analytics event type.
Values are page_view, search, or search_click.
POST _application/analytics/my_analytics_collection/event/search_click
{
"session": {
"id": "1797ca95-91c9-4e2e-b1bd-9c38e6f386a9"
},
"user": {
"id": "5f26f01a-bbee-4202-9298-81261067abbd"
},
"search":{
"query": "search term",
"results": {
"items": [
{
"document": {
"id": "123",
"index": "products"
}
}
],
"total_results": 10
},
"sort": {
"name": "relevance"
},
"search_application": "website"
},
"document":{
"id": "123",
"index": "products"
}
}resp = client.search_application.post_behavioral_analytics_event(
collection_name="my_analytics_collection",
event_type="search_click",
payload={
"session": {
"id": "1797ca95-91c9-4e2e-b1bd-9c38e6f386a9"
},
"user": {
"id": "5f26f01a-bbee-4202-9298-81261067abbd"
},
"search": {
"query": "search term",
"results": {
"items": [
{
"document": {
"id": "123",
"index": "products"
}
}
],
"total_results": 10
},
"sort": {
"name": "relevance"
},
"search_application": "website"
},
"document": {
"id": "123",
"index": "products"
}
},
)const response = await client.searchApplication.postBehavioralAnalyticsEvent({
collection_name: "my_analytics_collection",
event_type: "search_click",
payload: {
session: {
id: "1797ca95-91c9-4e2e-b1bd-9c38e6f386a9",
},
user: {
id: "5f26f01a-bbee-4202-9298-81261067abbd",
},
search: {
query: "search term",
results: {
items: [
{
document: {
id: "123",
index: "products",
},
},
],
total_results: 10,
},
sort: {
name: "relevance",
},
search_application: "website",
},
document: {
id: "123",
index: "products",
},
},
});response = client.search_application.post_behavioral_analytics_event(
collection_name: "my_analytics_collection",
event_type: "search_click",
body: {
"session": {
"id": "1797ca95-91c9-4e2e-b1bd-9c38e6f386a9"
},
"user": {
"id": "5f26f01a-bbee-4202-9298-81261067abbd"
},
"search": {
"query": "search term",
"results": {
"items": [
{
"document": {
"id": "123",
"index": "products"
}
}
],
"total_results": 10
},
"sort": {
"name": "relevance"
},
"search_application": "website"
},
"document": {
"id": "123",
"index": "products"
}
}
)$resp = $client->searchApplication()->postBehavioralAnalyticsEvent([
"collection_name" => "my_analytics_collection",
"event_type" => "search_click",
"body" => [
"session" => [
"id" => "1797ca95-91c9-4e2e-b1bd-9c38e6f386a9",
],
"user" => [
"id" => "5f26f01a-bbee-4202-9298-81261067abbd",
],
"search" => [
"query" => "search term",
"results" => [
"items" => array(
[
"document" => [
"id" => "123",
"index" => "products",
],
],
),
"total_results" => 10,
],
"sort" => [
"name" => "relevance",
],
"search_application" => "website",
],
"document" => [
"id" => "123",
"index" => "products",
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"session":{"id":"1797ca95-91c9-4e2e-b1bd-9c38e6f386a9"},"user":{"id":"5f26f01a-bbee-4202-9298-81261067abbd"},"search":{"query":"search term","results":{"items":[{"document":{"id":"123","index":"products"}}],"total_results":10},"sort":{"name":"relevance"},"search_application":"website"},"document":{"id":"123","index":"products"}}' "$ELASTICSEARCH_URL/_application/analytics/my_analytics_collection/event/search_click"client.searchApplication().postBehavioralAnalyticsEvent(p -> p
.collectionName("my_analytics_collection")
.eventType(EventType.SearchClick)
.payload(JsonData.fromJson("{\"session\":{\"id\":\"1797ca95-91c9-4e2e-b1bd-9c38e6f386a9\"},\"user\":{\"id\":\"5f26f01a-bbee-4202-9298-81261067abbd\"},\"search\":{\"query\":\"search term\",\"results\":{\"items\":[{\"document\":{\"id\":\"123\",\"index\":\"products\"}}],\"total_results\":10},\"sort\":{\"name\":\"relevance\"},\"search_application\":\"website\"},\"document\":{\"id\":\"123\",\"index\":\"products\"}}"))
);
{
"session": {
"id": "1797ca95-91c9-4e2e-b1bd-9c38e6f386a9"
},
"user": {
"id": "5f26f01a-bbee-4202-9298-81261067abbd"
},
"search":{
"query": "search term",
"results": {
"items": [
{
"document": {
"id": "123",
"index": "products"
}
}
],
"total_results": 10
},
"sort": {
"name": "relevance"
},
"search_application": "website"
},
"document":{
"id": "123",
"index": "products"
}
}All methods and paths for this operation:
Get the cluster's index aliases, including filter and routing information. This API does not return data stream aliases.
IMPORTANT: CAT APIs are only intended for human consumption using the command line or the Kibana console. They are not intended for use by applications. For application consumption, use the aliases API.
view_index_metadataA comma-separated list of aliases to retrieve. Supports wildcards (*). To retrieve all aliases, omit this parameter or use * or _all.
List of columns to appear in the response. Supports simple wildcards.
List of columns that determine how the table should be sorted.
Sorting defaults to ascending and can be changed by setting :asc
or :desc as a suffix to the column name.
The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
It supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, the request computes the list of selected nodes from the
local cluster state. If false the list of selected nodes are computed
from the cluster state of the master node. In both cases the coordinating
node will send requests for further information to each selected node.
GET _cat/aliases?format=json&v=true
resp = client.cat.aliases(
format="json",
v=True,
)const response = await client.cat.aliases({
format: "json",
v: "true",
});response = client.cat.aliases(
format: "json",
v: "true"
)$resp = $client->cat()->aliases([
"format" => "json",
"v" => "true",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cat/aliases?format=json&v=true"client.cat().aliases();
[
{
"alias": "alias1",
"index": "test1",
"filter": "-",
"routing.index": "-",
"routing.search": "-",
"is_write_index": "true"
},
{
"alias": "alias1",
"index": "test1",
"filter": "*",
"routing.index": "-",
"routing.search": "-",
"is_write_index": "true"
},
{
"alias": "alias3",
"index": "test1",
"filter": "-",
"routing.index": "1",
"routing.search": "1",
"is_write_index": "true"
},
{
"alias": "alias4",
"index": "test1",
"filter": "-",
"routing.index": "2",
"routing.search": "1,2",
"is_write_index": "true"
}
]curl \
--request GET 'http://api.example.com/_cat' \
--header "Authorization: $API_KEY"All methods and paths for this operation:
Get configuration and usage information about data frame analytics jobs.
IMPORTANT: CAT APIs are only intended for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, use the get data frame analytics jobs statistics API.
monitor_mlWhether to ignore if a wildcard expression matches no configs. (This includes _all string or when no configs have been specified)
The unit in which to display byte values
Values are b, kb, mb, gb, tb, or pb.
Comma-separated list of column names to display.
Supported values include:
assignment_explanation (or ae): Contains messages relating to the selection of a node.create_time (or ct, createTime): The time when the data frame analytics job was created.description (or d): A description of a job.dest_index (or di, destIndex): Name of the destination index.failure_reason (or fr, failureReason): Contains messages about the reason why a data frame analytics job failed.id: Identifier for the data frame analytics job.model_memory_limit (or mml, modelMemoryLimit): The approximate maximum amount of memory resources that are permitted for
the data frame analytics job.node.address (or na, nodeAddress): The network address of the node that the data frame analytics job is
assigned to.node.ephemeral_id (or ne, nodeEphemeralId): The ephemeral ID of the node that the data frame analytics job is assigned
to.node.id (or ni, nodeId): The unique identifier of the node that the data frame analytics job is
assigned to.node.name (or nn, nodeName): The name of the node that the data frame analytics job is assigned to.progress (or p): The progress report of the data frame analytics job by phase.source_index (or si, sourceIndex): Name of the source index.state (or s): Current state of the data frame analytics job.type (or t): The type of analysis that the data frame analytics job performs.version (or v): The Elasticsearch version number in which the data frame analytics job was
created.Comma-separated list of column names or column aliases used to sort the response.
Supported values include:
assignment_explanation (or ae): Contains messages relating to the selection of a node.create_time (or ct, createTime): The time when the data frame analytics job was created.description (or d): A description of a job.dest_index (or di, destIndex): Name of the destination index.failure_reason (or fr, failureReason): Contains messages about the reason why a data frame analytics job failed.id: Identifier for the data frame analytics job.model_memory_limit (or mml, modelMemoryLimit): The approximate maximum amount of memory resources that are permitted for
the data frame analytics job.node.address (or na, nodeAddress): The network address of the node that the data frame analytics job is
assigned to.node.ephemeral_id (or ne, nodeEphemeralId): The ephemeral ID of the node that the data frame analytics job is assigned
to.node.id (or ni, nodeId): The unique identifier of the node that the data frame analytics job is
assigned to.node.name (or nn, nodeName): The name of the node that the data frame analytics job is assigned to.progress (or p): The progress report of the data frame analytics job by phase.source_index (or si, sourceIndex): Name of the source index.state (or s): Current state of the data frame analytics job.type (or t): The type of analysis that the data frame analytics job performs.version (or v): The Elasticsearch version number in which the data frame analytics job was
created.Unit used to display time values.
Values are nanos, micros, ms, s, m, h, or d.
GET _cat/ml/data_frame/analytics?v=true&format=json
resp = client.cat.ml_data_frame_analytics(
v=True,
format="json",
)const response = await client.cat.mlDataFrameAnalytics({
v: "true",
format: "json",
});response = client.cat.ml_data_frame_analytics(
v: "true",
format: "json"
)$resp = $client->cat()->mlDataFrameAnalytics([
"v" => "true",
"format" => "json",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cat/ml/data_frame/analytics?v=true&format=json"client.cat().mlDataFrameAnalytics();
[
{
"id": "classifier_job_1",
"type": "classification",
"create_time": "2020-02-12T11:49:09.594Z",
"state": "stopped"
},
{
"id": "classifier_job_2",
"type": "classification",
"create_time": "2020-02-12T11:49:14.479Z",
"state": "stopped"
},
{
"id": "classifier_job_3",
"type": "classification",
"create_time": "2020-02-12T11:49:16.928Z",
"state": "stopped"
},
{
"id": "classifier_job_4",
"type": "classification",
"create_time": "2020-02-12T11:49:19.127Z",
"state": "stopped"
},
{
"id": "classifier_job_5",
"type": "classification",
"create_time": "2020-02-12T11:49:21.349Z",
"state": "stopped"
}
]All methods and paths for this operation:
Get configuration and usage information for anomaly detection jobs.
This API returns a maximum of 10,000 jobs.
If the Elasticsearch security features are enabled, you must have monitor_ml,
monitor, manage_ml, or manage cluster privileges to use this API.
IMPORTANT: CAT APIs are only intended for human consumption using the Kibana console or command line. They are not intended for use by applications. For application consumption, use the get anomaly detection job statistics API.
monitor_mlSpecifies what to do when the request:
_all string or no identifiers and there are no matches.If true, the API returns an empty jobs array when there are no matches and the subset of results when there
are partial matches. If false, the API returns a 404 status code when there are no matches or only partial
matches.
The unit used to display byte values.
Values are b, kb, mb, gb, tb, or pb.
Comma-separated list of column names to display.
Supported values include:
assignment_explanation (or ae): For open anomaly detection jobs only, contains messages relating to the
selection of a node to run the job.buckets.count (or bc, bucketsCount): The number of bucket results produced by the job.buckets.time.exp_avg (or btea, bucketsTimeExpAvg): Exponential moving average of all bucket processing times, in milliseconds.buckets.time.exp_avg_hour (or bteah, bucketsTimeExpAvgHour): Exponentially-weighted moving average of bucket processing times calculated
in a 1 hour time window, in milliseconds.buckets.time.max (or btmax, bucketsTimeMax): Maximum among all bucket processing times, in milliseconds.buckets.time.min (or btmin, bucketsTimeMin): Minimum among all bucket processing times, in milliseconds.buckets.time.total (or btt, bucketsTimeTotal): Sum of all bucket processing times, in milliseconds.data.buckets (or db, dataBuckets): The number of buckets processed.data.earliest_record (or der, dataEarliestRecord): The timestamp of the earliest chronologically input document.data.empty_buckets (or deb, dataEmptyBuckets): The number of buckets which did not contain any data.data.input_bytes (or dib, dataInputBytes): The number of bytes of input data posted to the anomaly detection job.data.input_fields (or dif, dataInputFields): The total number of fields in input documents posted to the anomaly
detection job. This count includes fields that are not used in the analysis.
However, be aware that if you are using a datafeed, it extracts only the
required fields from the documents it retrieves before posting them to the job.data.input_records (or dir, dataInputRecords): The number of input documents posted to the anomaly detection job.data.invalid_dates (or did, dataInvalidDates): The number of input documents with either a missing date field or a date
that could not be parsed.data.last (or dl, dataLast): The timestamp at which data was last analyzed, according to server time.data.last_empty_bucket (or dleb, dataLastEmptyBucket): The timestamp of the last bucket that did not contain any data.data.last_sparse_bucket (or dlsb, dataLastSparseBucket): The timestamp of the last bucket that was considered sparse.data.latest_record (or dlr, dataLatestRecord): The timestamp of the latest chronologically input document.data.missing_fields (or dmf, dataMissingFields): The number of input documents that are missing a field that the anomaly
detection job is configured to analyze. Input documents with missing fields
are still processed because it is possible that not all fields are missing.data.out_of_order_timestamps (or doot, dataOutOfOrderTimestamps): The number of input documents that have a timestamp chronologically
preceding the start of the current anomaly detection bucket offset by the
latency window. This information is applicable only when you provide data
to the anomaly detection job by using the post data API. These out of order
documents are discarded, since jobs require time series data to be in
ascending chronological order.data.processed_fields (or dpf, dataProcessedFields): The total number of fields in all the documents that have been processed by
the anomaly detection job. Only fields that are specified in the detector
configuration object contribute to this count. The timestamp is not
included in this count.data.processed_records (or dpr, dataProcessedRecords): The number of input documents that have been processed by the anomaly
detection job. This value includes documents with missing fields, since
they are nonetheless analyzed. If you use datafeeds and have aggregations
in your search query, the processed record count is the number of
aggregation results processed, not the number of Elasticsearch documents.data.sparse_buckets (or dsb, dataSparseBuckets): The number of buckets that contained few data points compared to the
expected number of data points.forecasts.memory.avg (or fmavg, forecastsMemoryAvg): The average memory usage in bytes for forecasts related to the anomaly
detection job.forecasts.memory.max (or fmmax, forecastsMemoryMax): The maximum memory usage in bytes for forecasts related to the anomaly
detection job.forecasts.memory.min (or fmmin, forecastsMemoryMin): The minimum memory usage in bytes for forecasts related to the anomaly
detection job.forecasts.memory.total (or fmt, forecastsMemoryTotal): The total memory usage in bytes for forecasts related to the anomaly
detection job.forecasts.records.avg (or fravg, forecastsRecordsAvg): The average number of model_forecast` documents written for forecasts
related to the anomaly detection job.forecasts.records.max (or frmax, forecastsRecordsMax): The maximum number of model_forecast documents written for forecasts
related to the anomaly detection job.forecasts.records.min (or frmin, forecastsRecordsMin): The minimum number of model_forecast documents written for forecasts
related to the anomaly detection job.forecasts.records.total (or frt, forecastsRecordsTotal): The total number of model_forecast documents written for forecasts
related to the anomaly detection job.forecasts.time.avg (or ftavg, forecastsTimeAvg): The average runtime in milliseconds for forecasts related to the anomaly
detection job.forecasts.time.max (or ftmax, forecastsTimeMax): The maximum runtime in milliseconds for forecasts related to the anomaly
detection job.forecasts.time.min (or ftmin, forecastsTimeMin): The minimum runtime in milliseconds for forecasts related to the anomaly
detection job.forecasts.time.total (or ftt, forecastsTimeTotal): The total runtime in milliseconds for forecasts related to the anomaly
detection job.forecasts.total (or ft, forecastsTotal): The number of individual forecasts currently available for the job.id: Identifier for the anomaly detection job.model.bucket_allocation_failures (or mbaf, modelBucketAllocationFailures): The number of buckets for which new entities in incoming data were not
processed due to insufficient model memory.model.by_fields (or mbf, modelByFields): The number of by field values that were analyzed by the models. This value
is cumulative for all detectors in the job.model.bytes (or mb, modelBytes): The number of bytes of memory used by the models. This is the maximum value
since the last time the model was persisted. If the job is closed, this
value indicates the latest size.model.bytes_exceeded (or mbe, modelBytesExceeded): The number of bytes over the high limit for memory usage at the last
allocation failure.model.categorization_status (or mcs, modelCategorizationStatus): The status of categorization for the job: ok or warn. If ok,
categorization is performing acceptably well (or not being used at all). If
warn, categorization is detecting a distribution of categories that
suggests the input data is inappropriate for categorization. Problems could
be that there is only one category, more than 90% of categories are rare,
the number of categories is greater than 50% of the number of categorized
documents, there are no frequently matched categories, or more than 50% of
categories are dead.model.categorized_doc_count (or mcdc, modelCategorizedDocCount): The number of documents that have had a field categorized.model.dead_category_count (or mdcc, modelDeadCategoryCount): The number of categories created by categorization that will never be
assigned again because another category’s definition makes it a superset of
the dead category. Dead categories are a side effect of the way
categorization has no prior training.model.failed_category_count (or mdcc, modelFailedCategoryCount): The number of times that categorization wanted to create a new category but
couldn’t because the job had hit its model memory limit. This count does
not track which specific categories failed to be created. Therefore, you
cannot use this value to determine the number of unique categories that
were missed.model.frequent_category_count (or mfcc, modelFrequentCategoryCount): The number of categories that match more than 1% of categorized documents.model.log_time (or mlt, modelLogTime): The timestamp when the model stats were gathered, according to server time.model.memory_limit (or mml, modelMemoryLimit): The timestamp when the model stats were gathered, according to server time.model.memory_status (or mms, modelMemoryStatus): The status of the mathematical models: ok, soft_limit, or hard_limit.
If ok, the models stayed below the configured value. If soft_limit, the
models used more than 60% of the configured memory limit and older unused
models will be pruned to free up space. Additionally, in categorization jobs
no further category examples will be stored. If hard_limit, the models
used more space than the configured memory limit. As a result, not all
incoming data was processed.model.over_fields (or mof, modelOverFields): The number of over field values that were analyzed by the models. This
value is cumulative for all detectors in the job.model.partition_fields (or mpf, modelPartitionFields): The number of partition field values that were analyzed by the models. This
value is cumulative for all detectors in the job.model.rare_category_count (or mrcc, modelRareCategoryCount): The number of categories that match just one categorized document.model.timestamp (or mt, modelTimestamp): The timestamp of the last record when the model stats were gathered.model.total_category_count (or mtcc, modelTotalCategoryCount): The number of categories created by categorization.node.address (or na, nodeAddress): The network address of the node that runs the job. This information is
available only for open jobs.node.ephemeral_id (or ne, nodeEphemeralId): The ephemeral ID of the node that runs the job. This information is
available only for open jobs.node.id (or ni, nodeId): The unique identifier of the node that runs the job. This information is
available only for open jobs.node.name (or nn, nodeName): The name of the node that runs the job. This information is available only
for open jobs.opened_time (or ot): For open jobs only, the elapsed time for which the job has been open.state (or s): The status of the anomaly detection job: closed, closing, failed,
opened, or opening. If closed, the job finished successfully with its
model state persisted. The job must be opened before it can accept further
data. If closing, the job close action is in progress and has not yet
completed. A closing job cannot accept further data. If failed, the job
did not finish successfully due to an error. This situation can occur due
to invalid input data, a fatal error occurring during the analysis, or an
external interaction such as the process being killed by the Linux out of
memory (OOM) killer. If the job had irrevocably failed, it must be force
closed and then deleted. If the datafeed can be corrected, the job can be
closed and then re-opened. If opened, the job is available to receive and
process data. If opening, the job open action is in progress and has not
yet completed.Comma-separated list of column names or column aliases used to sort the response.
Supported values include:
assignment_explanation (or ae): For open anomaly detection jobs only, contains messages relating to the
selection of a node to run the job.buckets.count (or bc, bucketsCount): The number of bucket results produced by the job.buckets.time.exp_avg (or btea, bucketsTimeExpAvg): Exponential moving average of all bucket processing times, in milliseconds.buckets.time.exp_avg_hour (or bteah, bucketsTimeExpAvgHour): Exponentially-weighted moving average of bucket processing times calculated
in a 1 hour time window, in milliseconds.buckets.time.max (or btmax, bucketsTimeMax): Maximum among all bucket processing times, in milliseconds.buckets.time.min (or btmin, bucketsTimeMin): Minimum among all bucket processing times, in milliseconds.buckets.time.total (or btt, bucketsTimeTotal): Sum of all bucket processing times, in milliseconds.data.buckets (or db, dataBuckets): The number of buckets processed.data.earliest_record (or der, dataEarliestRecord): The timestamp of the earliest chronologically input document.data.empty_buckets (or deb, dataEmptyBuckets): The number of buckets which did not contain any data.data.input_bytes (or dib, dataInputBytes): The number of bytes of input data posted to the anomaly detection job.data.input_fields (or dif, dataInputFields): The total number of fields in input documents posted to the anomaly
detection job. This count includes fields that are not used in the analysis.
However, be aware that if you are using a datafeed, it extracts only the
required fields from the documents it retrieves before posting them to the job.data.input_records (or dir, dataInputRecords): The number of input documents posted to the anomaly detection job.data.invalid_dates (or did, dataInvalidDates): The number of input documents with either a missing date field or a date
that could not be parsed.data.last (or dl, dataLast): The timestamp at which data was last analyzed, according to server time.data.last_empty_bucket (or dleb, dataLastEmptyBucket): The timestamp of the last bucket that did not contain any data.data.last_sparse_bucket (or dlsb, dataLastSparseBucket): The timestamp of the last bucket that was considered sparse.data.latest_record (or dlr, dataLatestRecord): The timestamp of the latest chronologically input document.data.missing_fields (or dmf, dataMissingFields): The number of input documents that are missing a field that the anomaly
detection job is configured to analyze. Input documents with missing fields
are still processed because it is possible that not all fields are missing.data.out_of_order_timestamps (or doot, dataOutOfOrderTimestamps): The number of input documents that have a timestamp chronologically
preceding the start of the current anomaly detection bucket offset by the
latency window. This information is applicable only when you provide data
to the anomaly detection job by using the post data API. These out of order
documents are discarded, since jobs require time series data to be in
ascending chronological order.data.processed_fields (or dpf, dataProcessedFields): The total number of fields in all the documents that have been processed by
the anomaly detection job. Only fields that are specified in the detector
configuration object contribute to this count. The timestamp is not
included in this count.data.processed_records (or dpr, dataProcessedRecords): The number of input documents that have been processed by the anomaly
detection job. This value includes documents with missing fields, since
they are nonetheless analyzed. If you use datafeeds and have aggregations
in your search query, the processed record count is the number of
aggregation results processed, not the number of Elasticsearch documents.data.sparse_buckets (or dsb, dataSparseBuckets): The number of buckets that contained few data points compared to the
expected number of data points.forecasts.memory.avg (or fmavg, forecastsMemoryAvg): The average memory usage in bytes for forecasts related to the anomaly
detection job.forecasts.memory.max (or fmmax, forecastsMemoryMax): The maximum memory usage in bytes for forecasts related to the anomaly
detection job.forecasts.memory.min (or fmmin, forecastsMemoryMin): The minimum memory usage in bytes for forecasts related to the anomaly
detection job.forecasts.memory.total (or fmt, forecastsMemoryTotal): The total memory usage in bytes for forecasts related to the anomaly
detection job.forecasts.records.avg (or fravg, forecastsRecordsAvg): The average number of model_forecast` documents written for forecasts
related to the anomaly detection job.forecasts.records.max (or frmax, forecastsRecordsMax): The maximum number of model_forecast documents written for forecasts
related to the anomaly detection job.forecasts.records.min (or frmin, forecastsRecordsMin): The minimum number of model_forecast documents written for forecasts
related to the anomaly detection job.forecasts.records.total (or frt, forecastsRecordsTotal): The total number of model_forecast documents written for forecasts
related to the anomaly detection job.forecasts.time.avg (or ftavg, forecastsTimeAvg): The average runtime in milliseconds for forecasts related to the anomaly
detection job.forecasts.time.max (or ftmax, forecastsTimeMax): The maximum runtime in milliseconds for forecasts related to the anomaly
detection job.forecasts.time.min (or ftmin, forecastsTimeMin): The minimum runtime in milliseconds for forecasts related to the anomaly
detection job.forecasts.time.total (or ftt, forecastsTimeTotal): The total runtime in milliseconds for forecasts related to the anomaly
detection job.forecasts.total (or ft, forecastsTotal): The number of individual forecasts currently available for the job.id: Identifier for the anomaly detection job.model.bucket_allocation_failures (or mbaf, modelBucketAllocationFailures): The number of buckets for which new entities in incoming data were not
processed due to insufficient model memory.model.by_fields (or mbf, modelByFields): The number of by field values that were analyzed by the models. This value
is cumulative for all detectors in the job.model.bytes (or mb, modelBytes): The number of bytes of memory used by the models. This is the maximum value
since the last time the model was persisted. If the job is closed, this
value indicates the latest size.model.bytes_exceeded (or mbe, modelBytesExceeded): The number of bytes over the high limit for memory usage at the last
allocation failure.model.categorization_status (or mcs, modelCategorizationStatus): The status of categorization for the job: ok or warn. If ok,
categorization is performing acceptably well (or not being used at all). If
warn, categorization is detecting a distribution of categories that
suggests the input data is inappropriate for categorization. Problems could
be that there is only one category, more than 90% of categories are rare,
the number of categories is greater than 50% of the number of categorized
documents, there are no frequently matched categories, or more than 50% of
categories are dead.model.categorized_doc_count (or mcdc, modelCategorizedDocCount): The number of documents that have had a field categorized.model.dead_category_count (or mdcc, modelDeadCategoryCount): The number of categories created by categorization that will never be
assigned again because another category’s definition makes it a superset of
the dead category. Dead categories are a side effect of the way
categorization has no prior training.model.failed_category_count (or mdcc, modelFailedCategoryCount): The number of times that categorization wanted to create a new category but
couldn’t because the job had hit its model memory limit. This count does
not track which specific categories failed to be created. Therefore, you
cannot use this value to determine the number of unique categories that
were missed.model.frequent_category_count (or mfcc, modelFrequentCategoryCount): The number of categories that match more than 1% of categorized documents.model.log_time (or mlt, modelLogTime): The timestamp when the model stats were gathered, according to server time.model.memory_limit (or mml, modelMemoryLimit): The timestamp when the model stats were gathered, according to server time.model.memory_status (or mms, modelMemoryStatus): The status of the mathematical models: ok, soft_limit, or hard_limit.
If ok, the models stayed below the configured value. If soft_limit, the
models used more than 60% of the configured memory limit and older unused
models will be pruned to free up space. Additionally, in categorization jobs
no further category examples will be stored. If hard_limit, the models
used more space than the configured memory limit. As a result, not all
incoming data was processed.model.over_fields (or mof, modelOverFields): The number of over field values that were analyzed by the models. This
value is cumulative for all detectors in the job.model.partition_fields (or mpf, modelPartitionFields): The number of partition field values that were analyzed by the models. This
value is cumulative for all detectors in the job.model.rare_category_count (or mrcc, modelRareCategoryCount): The number of categories that match just one categorized document.model.timestamp (or mt, modelTimestamp): The timestamp of the last record when the model stats were gathered.model.total_category_count (or mtcc, modelTotalCategoryCount): The number of categories created by categorization.node.address (or na, nodeAddress): The network address of the node that runs the job. This information is
available only for open jobs.node.ephemeral_id (or ne, nodeEphemeralId): The ephemeral ID of the node that runs the job. This information is
available only for open jobs.node.id (or ni, nodeId): The unique identifier of the node that runs the job. This information is
available only for open jobs.node.name (or nn, nodeName): The name of the node that runs the job. This information is available only
for open jobs.opened_time (or ot): For open jobs only, the elapsed time for which the job has been open.state (or s): The status of the anomaly detection job: closed, closing, failed,
opened, or opening. If closed, the job finished successfully with its
model state persisted. The job must be opened before it can accept further
data. If closing, the job close action is in progress and has not yet
completed. A closing job cannot accept further data. If failed, the job
did not finish successfully due to an error. This situation can occur due
to invalid input data, a fatal error occurring during the analysis, or an
external interaction such as the process being killed by the Linux out of
memory (OOM) killer. If the job had irrevocably failed, it must be force
closed and then deleted. If the datafeed can be corrected, the job can be
closed and then re-opened. If opened, the job is available to receive and
process data. If opening, the job open action is in progress and has not
yet completed.The unit used to display time values.
Values are nanos, micros, ms, s, m, h, or d.
GET _cat/ml/anomaly_detectors?h=id,s,dpr,mb&v=true&format=json
resp = client.cat.ml_jobs(
h="id,s,dpr,mb",
v=True,
format="json",
)const response = await client.cat.mlJobs({
h: "id,s,dpr,mb",
v: "true",
format: "json",
});response = client.cat.ml_jobs(
h: "id,s,dpr,mb",
v: "true",
format: "json"
)$resp = $client->cat()->mlJobs([
"h" => "id,s,dpr,mb",
"v" => "true",
"format" => "json",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cat/ml/anomaly_detectors?h=id,s,dpr,mb&v=true&format=json"client.cat().mlJobs();
[
{
"id": "high_sum_total_sales",
"s": "closed",
"dpr": "14022",
"mb": "1.5mb"
},
{
"id": "low_request_rate",
"s": "closed",
"dpr": "1216",
"mb": "40.5kb"
},
{
"id": "response_code_rates",
"s": "closed",
"dpr": "28146",
"mb": "132.7kb"
},
{
"id": "url_scanning",
"s": "closed",
"dpr": "28146",
"mb": "501.6kb"
}
]All methods and paths for this operation:
Get low-level information about the Lucene segments in index shards. For data streams, the API returns information about the backing indices. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the index segments API.
monitormonitorA comma-separated list of data streams, indices, and aliases used to limit the request.
Supports wildcards (*).
To target all data streams and indices, omit this parameter or use * or _all.
The unit used to display byte values.
Values are b, kb, mb, gb, tb, or pb.
A comma-separated list of columns names to display. It supports simple wildcards.
Supported values include:
index (or i, idx): The name of the index.shard (or s, sh): The name of the shard.prirep (or p, pr, primaryOrReplica): The shard type. Returned values are 'primary' or 'replica'.ip: IP address of the segment’s shard, such as '127.0.1.1'.segment: The name of the segment, such as '_0'. The segment name is derived from the segment generation and used internally to create file names in the directory of the shard.generation: Generation number, such as '0'. Elasticsearch increments this generation number for each segment written. Elasticsearch then uses this number to derive the segment name.docs.count: The number of documents as reported by Lucene. This excludes deleted documents and counts any nested documents separately from their parents. It also excludes documents which were indexed recently and do not yet belong to a segment.docs.deleted: The number of deleted documents as reported by Lucene, which may be higher or lower than the number of delete operations you have performed. This number excludes deletes that were performed recently and do not yet belong to a segment. Deleted documents are cleaned up by the automatic merge process if it makes sense to do so. Also, Elasticsearch creates extra deleted documents to internally track the recent history of operations on a shard.size: The disk space used by the segment, such as '50kb'.size.memory: The bytes of segment data stored in memory for efficient search, such as '1264'. A value of '-1' indicates Elasticsearch was unable to compute this number.committed: If 'true', the segments is synced to disk. Segments that are synced can survive a hard reboot. If 'false', the data from uncommitted segments is also stored in the transaction log so that Elasticsearch is able to replay changes on the next start.searchable: If 'true', the segment is searchable. If 'false', the segment has most likely been written to disk but needs a refresh to be searchable.version: The version of Lucene used to write the segment.compound: If 'true', the segment is stored in a compound file. This means Lucene merged all files from the segment in a single file to save file descriptors.id: The ID of the node, such as 'k0zy'.Values are index, i, idx, shard, s, sh, prirep, p, pr, primaryOrReplica, ip, segment, generation, docs.count, docs.deleted, size, size.memory, committed, searchable, version, compound, or id.
A comma-separated list of column names or aliases that determines the sort order.
Sorting defaults to ascending and can be changed by setting :asc
or :desc as a suffix to the column name.
If true, the request computes the list of selected nodes from the
local cluster state. If false the list of selected nodes are computed
from the cluster state of the master node. In both cases the coordinating
node will send requests for further information to each selected node.
Period to wait for a connection to the master node.
Values are -1 or 0.
GET /_cat/segments?v=true&format=json
resp = client.cat.segments(
v=True,
format="json",
)const response = await client.cat.segments({
v: "true",
format: "json",
});response = client.cat.segments(
v: "true",
format: "json"
)$resp = $client->cat()->segments([
"v" => "true",
"format" => "json",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cat/segments?v=true&format=json"client.cat().segments();
[
{
"index": "test",
"shard": "0",
"prirep": "p",
"ip": "127.0.0.1",
"segment": "_0",
"generation": "0",
"docs.count": "1",
"docs.deleted": "0",
"size": "3kb",
"size.memory": "0",
"committed": "false",
"searchable": "true",
"version": "9.12.0",
"compound": "true"
},
{
"index": "test1",
"shard": "0",
"prirep": "p",
"ip": "127.0.0.1",
"segment": "_0",
"generation": "0",
"docs.count": "1",
"docs.deleted": "0",
"size": "3kb",
"size.memory": "0",
"committed": "false",
"searchable": "true",
"version": "9.12.0",
"compound": "true"
}
]All methods and paths for this operation:
Get information about the shards in a cluster. For data streams, the API returns information about the backing indices. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications.
monitormonitorA comma-separated list of data streams, indices, and aliases used to limit the request.
Supports wildcards (*).
To target all data streams and indices, omit this parameter or use * or _all.
The unit used to display byte values.
Values are b, kb, mb, gb, tb, or pb.
List of columns to appear in the response. Supports simple wildcards.
Supported values include:
completion.size (or cs, completionSize): Size of completion. For example: 0b.dataset.size: Disk space used by the shard’s dataset, which may or may not be the size on
disk, but includes space used by the shard on object storage. Reported as a size value for example: 5kb.dense_vector.value_count (or dvc, denseVectorCount): Number of indexed dense vectors.docs (or d, dc): Number of documents in shard, for example: 25.fielddata.evictions (or fe, fielddataEvictions): Fielddata cache evictions, for example: 0.fielddata.memory_size (or fm, fielddataMemory): Used fielddata cache memory, for example: 0b.flush.total (or ft, flushTotal): Number of flushes, for example: 1.flush.total_time (or ftt, flushTotalTime): Time spent in flush, for example: 1.get.current (or gc, getCurrent): Number of current get operations, for example: 0.get.exists_time (or geti, getExistsTime): Time spent in successful gets, for example: 14ms.get.exists_total (or geto, getExistsTotal): Number of successful get operations, for example: 2.get.missing_time (or gmti, getMissingTime): Time spent in failed gets, for example: 0s.get.missing_total (or gmto, getMissingTotal): Number of failed get operations, for example: 1.get.time (or gti, getTime): Time spent in get, for example: 14ms.get.total (or gto, getTotal): Number of get operations, for example: 2.id: ID of the node, for example: k0zy.index (or i, idx): Name of the index.indexing.delete_current (or idc, indexingDeleteCurrent): Number of current deletion operations, for example: 0.indexing.delete_time (or idti, indexingDeleteTime): Time spent in deletions, for example: 2ms.indexing.delete_total (or idto, indexingDeleteTotal): Number of deletion operations, for example: 2.indexing.index_current (or iic, indexingIndexCurrent): Number of current indexing operations, for example: 0.indexing.index_failed_due_to_version_conflict (or iifvc, indexingIndexFailedDueToVersionConflict): Number of failed indexing operations due to version conflict, for example: 0.indexing.index_failed (or iif, indexingIndexFailed): Number of failed indexing operations, for example: 0.indexing.index_time (or iiti, indexingIndexTime): Time spent in indexing, such as for example: 134ms.indexing.index_total (or iito, indexingIndexTotal): Number of indexing operations, for example: 1.ip: IP address of the node, for example: 127.0.1.1.merges.current (or mc, mergesCurrent): Number of current merge operations, for example: 0.merges.current_docs (or mcd, mergesCurrentDocs): Number of current merging documents, for example: 0.merges.current_size (or mcs, mergesCurrentSize): Size of current merges, for example: 0b.merges.total (or mt, mergesTotal): Number of completed merge operations, for example: 0.merges.total_docs (or mtd, mergesTotalDocs): Number of merged documents, for example: 0.merges.total_size (or mts, mergesTotalSize): Size of current merges, for example: 0b.merges.total_time (or mtt, mergesTotalTime): Time spent merging documents, for example: 0s.node (or n): Node name, for example: I8hydUG.prirep (or p, pr, primaryOrReplica): Shard type. Returned values are primary or replica.query_cache.evictions (or qce, queryCacheEvictions): Query cache evictions, for example: 0.query_cache.memory_size (or qcm, queryCacheMemory): Used query cache memory, for example: 0b.recoverysource.type (or rs): Type of recovery source.refresh.time (or rti, refreshTime): Time spent in refreshes, for example: 91ms.refresh.total (or rto, refreshTotal): Number of refreshes, for example: 16.search.fetch_current (or sfc, searchFetchCurrent): Current fetch phase operations, for example: 0.search.fetch_time (or sfti, searchFetchTime): Time spent in fetch phase, for example: 37ms.search.fetch_total (or sfto, searchFetchTotal): Number of fetch operations, for example: 7.search.open_contexts (or so, searchOpenContexts): Open search contexts, for example: 0.search.query_current (or sqc, searchQueryCurrent): Current query phase operations, for example: 0.search.query_time (or sqti, searchQueryTime): Time spent in query phase, for example: 43ms.search.query_total (or sqto, searchQueryTotal): Number of query operations, for example: 9.search.scroll_current (or scc, searchScrollCurrent): Open scroll contexts, for example: 2.search.scroll_time (or scti, searchScrollTime): Time scroll contexts held open, for example: 2m.search.scroll_total (or scto, searchScrollTotal): Completed scroll contexts, for example: 1.segments.count (or sc, segmentsCount): Number of segments, for example: 4.segments.fixed_bitset_memory (or sfbm, fixedBitsetMemory): Memory used by fixed bit sets for nested object field types and type filters for types referred in join fields, for example: 1.0kb.segments.index_writer_memory (or siwm, segmentsIndexWriterMemory): Memory used by index writer, for example: 18mb.segments.memory (or sm, segmentsMemory): Memory used by segments, for example: 1.4kb.segments.version_map_memory (or svmm, segmentsVersionMapMemory): Memory used by version map, for example: 1.0kb.seq_no.global_checkpoint (or sqg, globalCheckpoint): Global checkpoint.seq_no.local_checkpoint (or sql, localCheckpoint): Local checkpoint.seq_no.max (or sqm, maxSeqNo): Maximum sequence number.shard (or s, sh): Name of the shard.dsparse_vector.value_count (or svc, sparseVectorCount): Number of indexed sparse vectors.state (or st): State of the shard. Returned values are:
INITIALIZING: The shard is recovering from a peer shard or gateway.RELOCATING: The shard is relocating.STARTED: The shard has started.UNASSIGNED: The shard is not assigned to any node.store (or sto): Disk space used by the shard, for example: 5kb.suggest.current (or suc, suggestCurrent): Number of current suggest operations, for example: 0.suggest.time (or suti, suggestTime): Time spent in suggest, for example: 0.suggest.total (or suto, suggestTotal): Number of suggest operations, for example: 0.sync_id: Sync ID of the shard.unassigned.at (or ua): Time at which the shard became unassigned in Coordinated Universal Time (UTC).unassigned.details (or ud): Details about why the shard became unassigned. This does not explain why the shard is currently unassigned. To understand why a shard
is not assigned, use the Cluster allocation explain API.unassigned.for (or uf): Time at which the shard was requested to be unassigned in Coordinated Universal Time (UTC).unassigned.reason (or ur): Indicates the reason for the last change to the state of this unassigned shard. This does not explain why the shard is currently unassigned.
To understand why a shard is not assigned, use the Cluster allocation explain API. Returned values include:
ALLOCATION_FAILED: Unassigned as a result of a failed allocation of the shard.CLUSTER_RECOVERED: Unassigned as a result of a full cluster recovery.DANGLING_INDEX_IMPORTED: Unassigned as a result of importing a dangling index.EXISTING_INDEX_RESTORED: Unassigned as a result of restoring into a closed index.FORCED_EMPTY_PRIMARY: The shard’s allocation was last modified by forcing an empty primary using the Cluster reroute API.INDEX_CLOSED: Unassigned because the index was closed.INDEX_CREATED: Unassigned as a result of an API creation of an index.INDEX_REOPENED: Unassigned as a result of opening a closed index.MANUAL_ALLOCATION: The shard’s allocation was last modified by the Cluster reroute API.NEW_INDEX_RESTORED: Unassigned as a result of restoring into a new index.NODE_LEFT: Unassigned as a result of the node hosting it leaving the cluster.NODE_RESTARTING: Similar to NODE_LEFT, except that the node was registered as restarting using the Node shutdown API.PRIMARY_FAILED: The shard was initializing as a replica, but the primary shard failed before the initialization completed.REALLOCATED_REPLICA: A better replica location is identified and causes the existing replica allocation to be cancelled.REINITIALIZED: When a shard moves from started back to initializing.REPLICA_ADDED: Unassigned as a result of explicit addition of a replica.REROUTE_CANCELLED: Unassigned as a result of explicit cancel reroute command.Values are completion.size, cs, completionSize, dataset.size, dense_vector.value_count, dvc, denseVectorCount, docs, d, dc, fielddata.evictions, fe, fielddataEvictions, fielddata.memory_size, fm, fielddataMemory, flush.total, ft, flushTotal, flush.total_time, ftt, flushTotalTime, get.current, gc, getCurrent, get.exists_time, geti, getExistsTime, get.exists_total, geto, getExistsTotal, get.missing_time, gmti, getMissingTime, get.missing_total, gmto, getMissingTotal, get.time, gti, getTime, get.total, gto, getTotal, id, index, i, idx, indexing.delete_current, idc, indexingDeleteCurrent, indexing.delete_time, idti, indexingDeleteTime, indexing.delete_total, idto, indexingDeleteTotal, indexing.index_current, iic, indexingIndexCurrent, indexing.index_failed_due_to_version_conflict, iifvc, indexingIndexFailedDueToVersionConflict, indexing.index_failed, iif, indexingIndexFailed, indexing.index_time, iiti, indexingIndexTime, indexing.index_total, iito, indexingIndexTotal, ip, merges.current, mc, mergesCurrent, merges.current_docs, mcd, mergesCurrentDocs, merges.current_size, mcs, mergesCurrentSize, merges.total, mt, mergesTotal, merges.total_docs, mtd, mergesTotalDocs, merges.total_size, mts, mergesTotalSize, merges.total_time, mtt, mergesTotalTime, node, n, prirep, p, pr, primaryOrReplica, query_cache.evictions, qce, queryCacheEvictions, query_cache.memory_size, qcm, queryCacheMemory, recoverysource.type, rs, refresh.time, rti, refreshTime, refresh.total, rto, refreshTotal, search.fetch_current, sfc, searchFetchCurrent, search.fetch_time, sfti, searchFetchTime, search.fetch_total, sfto, searchFetchTotal, search.open_contexts, so, searchOpenContexts, search.query_current, sqc, searchQueryCurrent, search.query_time, sqti, searchQueryTime, search.query_total, sqto, searchQueryTotal, search.scroll_current, scc, searchScrollCurrent, search.scroll_time, scti, searchScrollTime, search.scroll_total, scto, searchScrollTotal, segments.count, sc, segmentsCount, segments.fixed_bitset_memory, sfbm, fixedBitsetMemory, segments.index_writer_memory, siwm, segmentsIndexWriterMemory, segments.memory, sm, segmentsMemory, segments.version_map_memory, svmm, segmentsVersionMapMemory, seq_no.global_checkpoint, sqg, globalCheckpoint, seq_no.local_checkpoint, sql, localCheckpoint, seq_no.max, sqm, maxSeqNo, shard, s, sh, dsparse_vector.value_count, svc, sparseVectorCount, state, st, store, sto, suggest.current, suc, suggestCurrent, suggest.time, suti, suggestTime, suggest.total, suto, suggestTotal, sync_id, unassigned.at, ua, unassigned.details, ud, unassigned.for, uf, unassigned.reason, or ur.
A comma-separated list of column names or aliases that determines the sort order.
Sorting defaults to ascending and can be changed by setting :asc
or :desc as a suffix to the column name.
The period to wait for a connection to the master node.
Values are -1 or 0.
The unit used to display time values.
Values are nanos, micros, ms, s, m, h, or d.
GET _cat/shards?format=json
resp = client.cat.shards(
format="json",
)const response = await client.cat.shards({
format: "json",
});response = client.cat.shards(
format: "json"
)$resp = $client->cat()->shards([
"format" => "json",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cat/shards?format=json"client.cat().shards();
[
{
"index": "my-index-000001",
"shard": "0",
"prirep": "p",
"state": "STARTED",
"docs": "3014",
"store": "31.1mb",
"dataset": "249b",
"ip": "192.168.56.10",
"node": "H5dfFeA"
}
][
{
"index": "my-index-000001",
"shard": "0",
"prirep": "p",
"state": "STARTED",
"docs": "3014",
"store": "31.1mb",
"dataset": "249b",
"ip": "192.168.56.10",
"node": "H5dfFeA"
}
][
{
"index": "my-index-000001",
"shard": "0",
"prirep": "p",
"state": "RELOCATING",
"docs": "3014",
"store": "31.1mb",
"dataset": "249b",
"ip": "192.168.56.10",
"node": "H5dfFeA -> -> 192.168.56.30 bGG90GE"
}
][
{
"index": "my-index-000001",
"shard": "0",
"prirep": "p",
"state": "STARTED",
"docs": "3014",
"store": "31.1mb",
"dataset": "249b",
"ip": "192.168.56.10",
"node": "H5dfFeA"
},
{
"index": "my-index-000001",
"shard": "0",
"prirep": "r",
"state": "INITIALIZING",
"docs": "0",
"store": "14.3mb",
"dataset": "249b",
"ip": "192.168.56.30",
"node": "bGG90GE"
}
][
{
"index": "my-index-000001",
"shard": "0",
"prirep": "p",
"state": "STARTED",
"unassigned.reason": "3014 31.1mb 192.168.56.10 H5dfFeA"
},
{
"index": "my-index-000001",
"shard": "0",
"prirep": "r",
"state": "STARTED",
"unassigned.reason": "3014 31.1mb 192.168.56.30 bGG90GE"
},
{
"index": "my-index-000001",
"shard": "0",
"prirep": "r",
"state": "STARTED",
"unassigned.reason": "3014 31.1mb 192.168.56.20 I8hydUG"
},
{
"index": "my-index-000001",
"shard": "0",
"prirep": "r",
"state": "UNASSIGNED",
"unassigned.reason": "ALLOCATION_FAILED"
}
]All methods and paths for this operation:
Get information about the index templates in a cluster. You can use index templates to apply index settings and field mappings to new indices at creation. IMPORTANT: cat APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the get index template API.
monitorThe name of the template to return. Accepts wildcard expressions. If omitted, all templates are returned.
List of columns to appear in the response. Supports simple wildcards.
List of columns that determine how the table should be sorted.
Sorting defaults to ascending and can be changed by setting :asc
or :desc as a suffix to the column name.
If true, the request computes the list of selected nodes from the
local cluster state. If false the list of selected nodes are computed
from the cluster state of the master node. In both cases the coordinating
node will send requests for further information to each selected node.
Period to wait for a connection to the master node.
Values are -1 or 0.
GET _cat/templates/my-template-*?v=true&s=name&format=json
resp = client.cat.templates(
name="my-template-*",
v=True,
s="name",
format="json",
)const response = await client.cat.templates({
name: "my-template-*",
v: "true",
s: "name",
format: "json",
});response = client.cat.templates(
name: "my-template-*",
v: "true",
s: "name",
format: "json"
)$resp = $client->cat()->templates([
"name" => "my-template-*",
"v" => "true",
"s" => "name",
"format" => "json",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cat/templates/my-template-*?v=true&s=name&format=json"client.cat().templates();
[
{
"name": "my-template-0",
"index_patterns": "[te*]",
"order": "500",
"version": null,
"composed_of": "[]"
},
{
"name": "my-template-1",
"index_patterns": "[tea*]",
"order": "501",
"version": null,
"composed_of": "[]"
},
{
"name": "my-template-2",
"index_patterns": "[teak*]",
"order": "502",
"version": "7",
"composed_of": "[]"
}
]All methods and paths for this operation:
Get explanations for shard allocations in the cluster. For unassigned shards, it provides an explanation for why the shard is unassigned. For assigned shards, it provides an explanation for why the shard is remaining on its current node and has not moved or rebalanced to another node. This API can be very useful when attempting to diagnose why a shard is unassigned or why a shard continues to remain on its current node when you might expect otherwise.
If true, returns information about disk usage and shard sizes.
If true, returns YES decisions in explanation.
Period to wait for a connection to the master node.
Values are -1 or 0.
GET _cluster/allocation/explain
{
"index": "my-index-000001",
"shard": 0,
"primary": false,
"current_node": "my-node"
}resp = client.cluster.allocation_explain(
index="my-index-000001",
shard=0,
primary=False,
current_node="my-node",
)const response = await client.cluster.allocationExplain({
index: "my-index-000001",
shard: 0,
primary: false,
current_node: "my-node",
});response = client.cluster.allocation_explain(
body: {
"index": "my-index-000001",
"shard": 0,
"primary": false,
"current_node": "my-node"
}
)$resp = $client->cluster()->allocationExplain([
"body" => [
"index" => "my-index-000001",
"shard" => 0,
"primary" => false,
"current_node" => "my-node",
],
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"index":"my-index-000001","shard":0,"primary":false,"current_node":"my-node"}' "$ELASTICSEARCH_URL/_cluster/allocation/explain"client.cluster().allocationExplain(a -> a
.currentNode("my-node")
.index("my-index-000001")
.primary(false)
.shard(0)
);
{
"index": "my-index-000001",
"shard": 0,
"primary": false,
"current_node": "my-node"
}{
"index" : "my-index-000001",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "INDEX_CREATED",
"at" : "2017-01-04T18:08:16.600Z",
"last_allocation_status" : "no"
},
"can_allocate" : "no",
"allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"node_allocation_decisions" : [
{
"node_id" : "8qt2rY-pT6KNZB3-hGfLnw",
"node_name" : "node-0",
"transport_address" : "127.0.0.1:9401",
"roles" : ["data", "data_cold", "data_content", "data_frozen", "data_hot", "data_warm", "ingest", "master", "ml", "remote_cluster_client", "transform"],
"node_attributes" : {},
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "filter",
"decision" : "NO",
"explanation" : "node does not match index setting [index.routing.allocation.include] filters [_name:\"nonexistent_node\"]"
}
]
}
]
}{
"index" : "my-index-000001",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"at" : "2017-01-04T18:03:28.464Z",
"failed shard on node [mEKjwwzLT1yJVb8UxT6anw]: failed recovery, failure RecoveryFailedException",
"reason": "ALLOCATION_FAILED",
"failed_allocation_attempts": 5,
"last_allocation_status": "no",
},
"can_allocate": "no",
"allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "3sULLVJrRneSg0EfBB-2Ew",
"node_name" : "node_t0",
"transport_address" : "127.0.0.1:9400",
"roles" : ["data_content", "data_hot"],
"node_decision" : "no",
"store" : {
"matching_size" : "4.2kb",
"matching_size_in_bytes" : 4325
},
"deciders" : [
{
"decider": "max_retry",
"decision" : "NO",
"explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2024-07-30T21:04:12.166Z], failed_attempts[5], failed_nodes[[mEKjwwzLT1yJVb8UxT6anw]], delayed=false, details[failed shard on node [mEKjwwzLT1yJVb8UxT6anw]: failed recovery, failure RecoveryFailedException], allocation_status[deciders_no]]]"
}
]
}
]
}Update the cluster voting config exclusions by node IDs or node names. By default, if there are more than three master-eligible nodes in the cluster and you remove fewer than half of the master-eligible nodes in the cluster at once, the voting configuration automatically shrinks. If you want to shrink the voting configuration to contain fewer than three nodes or to remove half or more of the master-eligible nodes in the cluster at once, use this API to remove departing nodes from the voting configuration manually. The API adds an entry for each specified node to the cluster’s voting configuration exclusions list. It then waits until the cluster has reconfigured its voting configuration to exclude the specified nodes.
Clusters should have no voting configuration exclusions in normal operation.
Once the excluded nodes have stopped, clear the voting configuration exclusions with DELETE /_cluster/voting_config_exclusions.
This API waits for the nodes to be fully removed from the cluster before it returns.
If your cluster has voting configuration exclusions for nodes that you no longer intend to remove, use DELETE /_cluster/voting_config_exclusions?wait_for_removal=false to clear the voting configuration exclusions without waiting for the nodes to leave the cluster.
A response to POST /_cluster/voting_config_exclusions with an HTTP status code of 200 OK guarantees that the node has been removed from the voting configuration and will not be reinstated until the voting configuration exclusions are cleared by calling DELETE /_cluster/voting_config_exclusions.
If the call to POST /_cluster/voting_config_exclusions fails or returns a response with an HTTP status code other than 200 OK then the node may not have been removed from the voting configuration.
In that case, you may safely retry the call.
NOTE: Voting exclusions are required only when you remove at least half of the master-eligible nodes from a cluster in a short time period. They are not required when removing master-ineligible nodes or when removing fewer than half of the master-eligible nodes.
A comma-separated list of the names of the nodes to exclude from the voting configuration. If specified, you may not also specify node_ids.
A comma-separated list of the persistent ids of the nodes to exclude from the voting configuration. If specified, you may not also specify node_names.
Period to wait for a connection to the master node.
Values are -1 or 0.
When adding a voting configuration exclusion, the API waits for the specified nodes to be excluded from the voting configuration before returning. If the timeout expires before the appropriate condition is satisfied, the request fails and returns an error.
Values are -1 or 0.
curl \
--request POST 'http://api.example.com/_cluster/voting_config_exclusions' \
--header "Authorization: $API_KEY"By default, it returns only settings that have been explicitly defined.
monitorIf true, returns settings in flat format.
If true, also returns default values for all other cluster settings, reflecting the values
in the elasticsearch.yml file of one of the nodes in the cluster. If the nodes in your
cluster do not all have the same values in their elasticsearch.yml config files then the
values returned by this API may vary from invocation to invocation and may not reflect the
values that Elasticsearch uses in all situations. Use the GET _nodes/settings API to
fetch the settings for each individual node in your cluster.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
GET /_cluster/settings?filter_path=persistent.cluster.remote
resp = client.cluster.get_settings(
filter_path="persistent.cluster.remote",
)const response = await client.cluster.getSettings({
filter_path: "persistent.cluster.remote",
});response = client.cluster.get_settings(
filter_path: "persistent.cluster.remote"
)$resp = $client->cluster()->getSettings([
"filter_path" => "persistent.cluster.remote",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cluster/settings?filter_path=persistent.cluster.remote"All methods and paths for this operation:
You can also use the API to get the health status of only specified data streams and indices. For data streams, the API retrieves the health status of the stream’s backing indices.
The cluster health status is: green, yellow or red. On the shard level, a red status indicates that the specific shard is not allocated in the cluster. Yellow means that the primary shard is allocated but replicas are not. Green means that all shards are allocated. The index level status is controlled by the worst shard status.
One of the main benefits of the API is the ability to wait until the cluster reaches a certain high watermark health level. The cluster status is controlled by the worst index status.
monitor,manageComma-separated list of data streams, indices, and index aliases used to limit the request. Wildcard expressions (*) are supported. To target all data streams and indices in a cluster, omit this parameter or use _all or *.
Whether to expand wildcard expression to concrete indices that are open, closed or both.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Can be one of cluster, indices or shards. Controls the details level of the health information returned.
Values are cluster, indices, or shards.
If true, the request retrieves information from the local node only. Defaults to false, which means information is retrieved from the master node.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
A number controlling to how many active shards to wait for, all to wait for all shards in the cluster to be active, or 0 to not wait.
Values are all or index-setting.
Can be one of immediate, urgent, high, normal, low, languid. Wait until all currently queued events with the given priority are processed.
Values are immediate, urgent, high, normal, low, or languid.
The request waits until the specified number N of nodes is available. It also accepts >=N, <=N, >N and <N. Alternatively, it is possible to use ge(N), le(N), gt(N) and lt(N) notation.
A boolean value which controls whether to wait (until the timeout provided) for the cluster to have no shard initializations. Defaults to false, which means it will not wait for initializing shards.
A boolean value which controls whether to wait (until the timeout provided) for the cluster to have no shard relocations. Defaults to false, which means it will not wait for relocating shards.
One of green, yellow or red. Will wait (until the timeout provided) until the status of the cluster changes to the one provided or better, i.e. green > yellow > red. By default, will not wait for any status.
Supported values include:
green (or GREEN): All shards are assigned.yellow (or YELLOW): All primary shards are assigned, but one or more replica shards are unassigned. If a node in the cluster fails, some data could be unavailable until that node is repaired.red (or RED): One or more primary shards are unassigned, so some data is unavailable. This can occur briefly during cluster startup as primary shards are assigned.unknownunavailableValues are green, GREEN, yellow, YELLOW, red, RED, unknown, or unavailable.
GET _cluster/health
resp = client.cluster.health()const response = await client.cluster.health();response = client.cluster.health$resp = $client->cluster()->health();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cluster/health"client.cluster().health(h -> h);
{
"cluster_name" : "testcluster",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 50.0
}Returns basic information about the cluster.
GET /_info/_all
resp = client.cluster.info(
target="_all",
)const response = await client.cluster.info({
target: "_all",
});response = client.cluster.info(
target: "_all"
)$resp = $client->cluster()->info([
"target" => "_all",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_info/_all"client.cluster().info(i -> i
.target("_all")
);
All methods and paths for this operation:
Get comprehensive information about the state of the cluster.
The cluster state is an internal data structure which keeps track of a variety of information needed by every node, including the identity and attributes of the other nodes in the cluster; cluster-wide settings; index metadata, including the mapping and settings for each index; the location and status of every shard copy in the cluster.
The elected master node ensures that every node in the cluster has a copy of the same cluster state. This API lets you retrieve a representation of this internal state for debugging or diagnostic purposes. You may need to consult the Elasticsearch source code to determine the precise meaning of the response.
By default the API will route requests to the elected master node since this node is the authoritative source of cluster states.
You can also retrieve the cluster state held on the node handling the API request by adding the ?local=true query parameter.
Elasticsearch may need to expend significant effort to compute a response to this API in larger clusters, and the response may comprise a very large quantity of data. If you use this API repeatedly, your cluster may become unstable.
WARNING: The response is a representation of an internal data structure. Its format is not subject to the same compatibility guarantees as other more stable APIs and may change from version to version. Do not query this API using external monitoring tools. Instead, obtain the information you require using other more stable cluster APIs.
monitor,manageLimit the information returned to the specified metrics
A comma-separated list of index names; use _all or empty string to perform the operation on all indices
Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)
Whether to expand wildcard expression to concrete indices that are open, closed or both.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Return settings in flat format (default: false)
Return local information, do not retrieve the state from master node (default: false)
Specify timeout for connection to master
Values are -1 or 0.
Wait for the metadata version to be equal or greater than the specified metadata version
The maximum time to wait for wait_for_metadata_version before timing out
Values are -1 or 0.
GET /_cluster/state?filter_path=metadata.cluster_coordination.last_committed_config
resp = client.cluster.state(
filter_path="metadata.cluster_coordination.last_committed_config",
)const response = await client.cluster.state({
filter_path: "metadata.cluster_coordination.last_committed_config",
});response = client.cluster.state(
filter_path: "metadata.cluster_coordination.last_committed_config"
)$resp = $client->cluster()->state([
"filter_path" => "metadata.cluster_coordination.last_committed_config",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cluster/state?filter_path=metadata.cluster_coordination.last_committed_config"All methods and paths for this operation:
Get a breakdown of the hot threads on each selected node in the cluster. The output is plain text with a breakdown of the top hot threads for each node.
monitor,manageIf true, known idle threads (e.g. waiting in a socket select, or to get a task from an empty queue) are filtered out.
The interval to do the second sampling of threads.
Values are -1 or 0.
Number of samples of thread stacktrace.
Specifies the number of hot threads to provide information for.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
The type to sample.
Values are cpu, wait, block, gpu, or mem.
The sort order for 'cpu' type (default: total)
Values are cpu, wait, block, gpu, or mem.
GET /_nodes/hot_threads
resp = client.nodes.hot_threads()const response = await client.nodes.hotThreads();response = client.nodes.hot_threads$resp = $client->nodes()->hotThreads();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_nodes/hot_threads"client.nodes().hotThreads(h -> h);
All methods and paths for this operation:
Secure settings are stored in an on-disk keystore. Certain of these settings are reloadable. That is, you can change them on disk and reload them without restarting any nodes in the cluster. When you have updated reloadable secure settings in your keystore, you can use this API to reload those settings on each node.
When the Elasticsearch keystore is password protected and not simply obfuscated, you must provide the password for the keystore when you reload the secure settings. Reloading the settings for the whole cluster assumes that the keystores for all nodes are protected with the same password; this method is allowed only when inter-node communications are encrypted. Alternatively, you can reload the secure settings on each node by locally accessing the API and passing the node-specific Elasticsearch keystore password.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
POST _nodes/reload_secure_settings
{
"secure_settings_password": "keystore-password"
}resp = client.nodes.reload_secure_settings(
secure_settings_password="keystore-password",
)const response = await client.nodes.reloadSecureSettings({
secure_settings_password: "keystore-password",
});response = client.nodes.reload_secure_settings(
body: {
"secure_settings_password": "keystore-password"
}
)$resp = $client->nodes()->reloadSecureSettings([
"body" => [
"secure_settings_password" => "keystore-password",
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"secure_settings_password":"keystore-password"}' "$ELASTICSEARCH_URL/_nodes/reload_secure_settings"client.nodes().reloadSecureSettings(r -> r
.secureSettingsPassword("keystore-password")
);
{
"secure_settings_password": "keystore-password"
}{
"_nodes": {
"total": 1,
"successful": 1,
"failed": 0
},
"cluster_name": "my_cluster",
"nodes": {
"pQHNt5rXTTWNvUgOrdynKg": {
"name": "node-0"
}
}
}All methods and paths for this operation:
Get statistics for nodes in a cluster. By default, all stats are returned. You can limit the returned information by using metrics.
monitor,manageComma-separated list of node IDs or names used to limit returned information.
Limit the information returned to the specified metrics
Limit the information returned for indices metric to the specific index metrics. It can be used only if indices (or all) metric is specified.
Comma-separated list or wildcard expressions of fields to include in fielddata and suggest statistics.
Comma-separated list or wildcard expressions of fields to include in fielddata statistics.
Comma-separated list or wildcard expressions of fields to include in the statistics.
Comma-separated list of search groups to include in the search statistics.
If true, the call reports the aggregated disk usage of each one of the Lucene index files (only applies if segment stats are requested).
Indicates whether statistics are aggregated at the cluster, index, or shard level.
Values are cluster, indices, or shards.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
A comma-separated list of document types for the indexing index metric.
If true, the response includes information from segments that are not loaded into memory.
GET _nodes/stats/process?filter_path=**.max_file_descriptors
resp = client.nodes.stats(
metric="process",
filter_path="**.max_file_descriptors",
)const response = await client.nodes.stats({
metric: "process",
filter_path: "**.max_file_descriptors",
});response = client.nodes.stats(
metric: "process",
filter_path: "**.max_file_descriptors"
)$resp = $client->nodes()->stats([
"metric" => "process",
"filter_path" => "**.max_file_descriptors",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_nodes/stats/process?filter_path=**.max_file_descriptors"A comma-separated list of node IDs or names to limit the returned information; use _local to return information from the node you're connecting to, leave empty to get information from all nodes
Limits the information returned to the specific metrics.
A comma-separated list of the following options: _all, rest_actions.
GET _nodes/usage
resp = client.nodes.usage()const response = await client.nodes.usage();response = client.nodes.usage$resp = $client->nodes()->usage();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_nodes/usage"client.nodes().usage(u -> u);
All methods and paths for this operation:
Get a report with the health status of an Elasticsearch cluster. The report contains a list of indicators that compose Elasticsearch functionality.
Each indicator has a health status of: green, unknown, yellow or red. The indicator will provide an explanation and metadata describing the reason for its current health status.
The cluster’s status is controlled by the worst indicator status.
In the event that an indicator’s status is non-green, a list of impacts may be present in the indicator result which detail the functionalities that are negatively affected by the health issue. Each impact carries with it a severity level, an area of the system that is affected, and a simple description of the impact on the system.
Some health indicators can determine the root cause of a health problem and prescribe a set of steps that can be performed in order to improve the health of the system. The root cause and remediation steps are encapsulated in a diagnosis. A diagnosis contains a cause detailing a root cause analysis, an action containing a brief description of the steps to take to fix the problem, the list of affected resources (if applicable), and a detailed step-by-step troubleshooting guide to fix the diagnosed problem.
NOTE: The health indicators perform root cause analysis of non-green health statuses. This can be computationally expensive when called frequently. When setting up automated polling of the API for health status, set verbose to false to disable the more expensive analysis logic.
GET _health_report
resp = client.health_report()const response = await client.healthReport();response = client.health_report$resp = $client->healthReport();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_health_report"client.healthReport(h -> h);
The connector and sync jobs APIs provide a convenient way to create and manage Elastic connectors and sync jobs in an internal index. Connectors are Elasticsearch integrations that bring content from third-party data sources, which can be deployed on Elastic Cloud or hosted on your own infrastructure:
This API provides an alternative to relying solely on Kibana UI for connector and sync job management. The API comes with a set of validations and assertions to ensure that the state representation in the internal index remains valid.
Update the last_seen field in the connector and set it to the current timestamp.
PUT _connector/my-connector/_check_in
resp = client.connector.check_in(
connector_id="my-connector",
)const response = await client.connector.checkIn({
connector_id: "my-connector",
});response = client.connector.check_in(
connector_id: "my-connector"
)$resp = $client->connector()->checkIn([
"connector_id" => "my-connector",
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/my-connector/_check_in"client.connector().checkIn(c -> c
.connectorId("my-connector")
);
{
"result": "updated"
}GET _connector/my-connector-id
resp = client.connector.get(
connector_id="my-connector-id",
)const response = await client.connector.get({
connector_id: "my-connector-id",
});response = client.connector.get(
connector_id: "my-connector-id"
)$resp = $client->connector()->get([
"connector_id" => "my-connector-id",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/my-connector-id"client.connector().get(g -> g
.connectorId("my-connector-id")
);
Removes a connector and associated sync jobs. This is a destructive action that is not recoverable. NOTE: This action doesn’t delete any API keys, ingest pipelines, or data indices associated with the connector. These need to be removed manually.
DELETE _connector/my-connector-id&delete_sync_jobs=true
resp = client.connector.delete(
connector_id="my-connector-id&delete_sync_jobs=true",
)const response = await client.connector.delete({
connector_id: "my-connector-id&delete_sync_jobs=true",
});response = client.connector.delete(
connector_id: "my-connector-id&delete_sync_jobs=true"
)$resp = $client->connector()->delete([
"connector_id" => "my-connector-id&delete_sync_jobs=true",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/my-connector-id&delete_sync_jobs=true"client.connector().delete(d -> d
.connectorId("my-connector-id&delete_sync_jobs=true")
);
{
"acknowledged": true
}Connectors are Elasticsearch integrations that bring content from third-party data sources, which can be deployed on Elastic Cloud or hosted on your own infrastructure. Elastic managed connectors (Native connectors) are a managed service on Elastic Cloud. Self-managed connectors (Connector clients) are self-managed on your infrastructure.
curl \
--request POST 'http://api.example.com/_connector' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '{"description":"string","index_name":"string","is_native":true,"language":"string","name":"string","service_type":"string"}'This action updates the job status to in_progress and sets the last_seen and started_at timestamps to the current time.
Additionally, it can set the sync_cursor property for the sync job.
This API is not intended for direct connector management by users. It supports the implementation of services that utilize the connector protocol to communicate with Elasticsearch.
To sync data using self-managed connectors, you need to deploy the Elastic connector service on your own infrastructure. This service runs automatically on Elastic Cloud for Elastic managed connectors.
PUT _connector/_sync_job/my-connector-sync-job-id/_claim
{
"worker_hostname": "some-machine"
}resp = client.connector.sync_job_claim(
connector_sync_job_id="my-connector-sync-job-id",
worker_hostname="some-machine",
)const response = await client.connector.syncJobClaim({
connector_sync_job_id: "my-connector-sync-job-id",
worker_hostname: "some-machine",
});response = client.connector.sync_job_claim(
connector_sync_job_id: "my-connector-sync-job-id",
body: {
"worker_hostname": "some-machine"
}
)$resp = $client->connector()->syncJobClaim([
"connector_sync_job_id" => "my-connector-sync-job-id",
"body" => [
"worker_hostname" => "some-machine",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"worker_hostname":"some-machine"}' "$ELASTICSEARCH_URL/_connector/_sync_job/my-connector-sync-job-id/_claim"client.connector().syncJobClaim(s -> s
.connectorSyncJobId("my-connector-sync-job-id")
.workerHostname("some-machine")
);
{
"worker_hostname": "some-machine"
}GET _connector/_sync_job/my-connector-sync-job
resp = client.connector.sync_job_get(
connector_sync_job_id="my-connector-sync-job",
)const response = await client.connector.syncJobGet({
connector_sync_job_id: "my-connector-sync-job",
});response = client.connector.sync_job_get(
connector_sync_job_id: "my-connector-sync-job"
)$resp = $client->connector()->syncJobGet([
"connector_sync_job_id" => "my-connector-sync-job",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/_sync_job/my-connector-sync-job"client.connector().syncJobGet(s -> s
.connectorSyncJobId("my-connector-sync-job")
);
Get information about all stored connector sync jobs listed by their creation date in ascending order.
Starting offset (default: 0)
Specifies a max number of results to get
A sync job status to fetch connector sync jobs for
Values are canceling, canceled, completed, error, in_progress, pending, or suspended.
A connector id to fetch connector sync jobs for
A comma-separated list of job types to fetch the sync jobs for
Supported values include: full, incremental, access_control
Values are full, incremental, or access_control.
GET _connector/_sync_job?connector_id=my-connector-id&size=1
resp = client.connector.sync_job_list(
connector_id="my-connector-id",
size="1",
)const response = await client.connector.syncJobList({
connector_id: "my-connector-id",
size: 1,
});response = client.connector.sync_job_list(
connector_id: "my-connector-id",
size: "1"
)$resp = $client->connector()->syncJobList([
"connector_id" => "my-connector-id",
"size" => "1",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/_sync_job?connector_id=my-connector-id&size=1"client.connector().syncJobList(s -> s
.connectorId("my-connector-id")
.size(1)
);
Update the api_key_id and api_key_secret_id fields of a connector.
You can specify the ID of the API key used for authorization and the ID of the connector secret where the API key is stored.
The connector secret ID is required only for Elastic managed (native) connectors.
Self-managed connectors (connector clients) do not use this field.
PUT _connector/my-connector/_api_key_id
{
"api_key_id": "my-api-key-id",
"api_key_secret_id": "my-connector-secret-id"
}resp = client.connector.update_api_key_id(
connector_id="my-connector",
api_key_id="my-api-key-id",
api_key_secret_id="my-connector-secret-id",
)const response = await client.connector.updateApiKeyId({
connector_id: "my-connector",
api_key_id: "my-api-key-id",
api_key_secret_id: "my-connector-secret-id",
});response = client.connector.update_api_key_id(
connector_id: "my-connector",
body: {
"api_key_id": "my-api-key-id",
"api_key_secret_id": "my-connector-secret-id"
}
)$resp = $client->connector()->updateApiKeyId([
"connector_id" => "my-connector",
"body" => [
"api_key_id" => "my-api-key-id",
"api_key_secret_id" => "my-connector-secret-id",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"api_key_id":"my-api-key-id","api_key_secret_id":"my-connector-secret-id"}' "$ELASTICSEARCH_URL/_connector/my-connector/_api_key_id"client.connector().updateApiKeyId(u -> u
.apiKeyId("my-api-key-id")
.apiKeySecretId("my-connector-secret-id")
.connectorId("my-connector")
);
{
"api_key_id": "my-api-key-id",
"api_key_secret_id": "my-connector-secret-id"
}{
"result": "updated"
}Set the error field for the connector. If the error provided in the request body is non-null, the connector’s status is updated to error. Otherwise, if the error is reset to null, the connector status is updated to connected.
PUT _connector/my-connector/_error
{
"error": "Houston, we have a problem!"
}resp = client.connector.update_error(
connector_id="my-connector",
error="Houston, we have a problem!",
)const response = await client.connector.updateError({
connector_id: "my-connector",
error: "Houston, we have a problem!",
});response = client.connector.update_error(
connector_id: "my-connector",
body: {
"error": "Houston, we have a problem!"
}
)$resp = $client->connector()->updateError([
"connector_id" => "my-connector",
"body" => [
"error" => "Houston, we have a problem!",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"error":"Houston, we have a problem!"}' "$ELASTICSEARCH_URL/_connector/my-connector/_error"client.connector().updateError(u -> u
.connectorId("my-connector")
.error("Houston, we have a problem!")
);
{
"error": "Houston, we have a problem!"
}{
"result": "updated"
}PUT _connector/my-connector/_name
{
"name": "Custom connector",
"description": "This is my customized connector"
}resp = client.connector.update_name(
connector_id="my-connector",
name="Custom connector",
description="This is my customized connector",
)const response = await client.connector.updateName({
connector_id: "my-connector",
name: "Custom connector",
description: "This is my customized connector",
});response = client.connector.update_name(
connector_id: "my-connector",
body: {
"name": "Custom connector",
"description": "This is my customized connector"
}
)$resp = $client->connector()->updateName([
"connector_id" => "my-connector",
"body" => [
"name" => "Custom connector",
"description" => "This is my customized connector",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"name":"Custom connector","description":"This is my customized connector"}' "$ELASTICSEARCH_URL/_connector/my-connector/_name"client.connector().updateName(u -> u
.connectorId("my-connector")
.description("This is my customized connector")
.name("Custom connector")
);
{
"name": "Custom connector",
"description": "This is my customized connector"
}{
"result": "updated"
}All methods and paths for this operation:
Get cross-cluster replication auto-follow patterns.
manage_ccrThe auto-follow pattern collection that you want to retrieve. If you do not specify a name, the API returns information for all collections.
GET /_ccr/auto_follow/my_auto_follow_pattern
resp = client.ccr.get_auto_follow_pattern(
name="my_auto_follow_pattern",
)const response = await client.ccr.getAutoFollowPattern({
name: "my_auto_follow_pattern",
});response = client.ccr.get_auto_follow_pattern(
name: "my_auto_follow_pattern"
)$resp = $client->ccr()->getAutoFollowPattern([
"name" => "my_auto_follow_pattern",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_ccr/auto_follow/my_auto_follow_pattern"client.ccr().getAutoFollowPattern(g -> g
.name("my_auto_follow_pattern")
);
{
"patterns": [
{
"name": "my_auto_follow_pattern",
"pattern": {
"active": true,
"remote_cluster" : "remote_cluster",
"leader_index_patterns" :
[
"leader_index*"
],
"leader_index_exclusion_patterns":
[
"leader_index_001"
],
"follow_index_pattern" : "{{leader_index}}-follower"
}
}
]
}Create a cross-cluster replication follower index that follows a specific leader index. When the API returns, the follower index exists and cross-cluster replication starts replicating operations from the leader index to the follower index.
Period to wait for a connection to the master node.
Values are -1 or 0.
Specifies the number of shards to wait on being active before responding. This defaults to waiting on none of the shards to be active. A shard must be restored from the leader index before being active. Restoring a follower shard requires transferring all the remote Lucene segment files to the follower index.
Values are all or index-setting.
If the leader index is part of a data stream, the name to which the local data stream for the followed index should be renamed.
The maximum number of outstanding reads requests from the remote cluster.
The maximum number of outstanding write requests on the follower.
The maximum number of operations to pull per read from the remote cluster.
A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and
d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.
The maximum number of operations that can be queued for writing. When this limit is reached, reads from the remote cluster will be deferred until the number of queued operations goes below the limit.
The maximum number of operations per bulk write request executed on the follower.
A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and
d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.
The remote cluster containing the leader index.
PUT /follower_index/_ccr/follow?wait_for_active_shards=1
{
"remote_cluster" : "remote_cluster",
"leader_index" : "leader_index",
"settings": {
"index.number_of_replicas": 0
},
"max_read_request_operation_count" : 1024,
"max_outstanding_read_requests" : 16,
"max_read_request_size" : "1024k",
"max_write_request_operation_count" : 32768,
"max_write_request_size" : "16k",
"max_outstanding_write_requests" : 8,
"max_write_buffer_count" : 512,
"max_write_buffer_size" : "512k",
"max_retry_delay" : "10s",
"read_poll_timeout" : "30s"
}resp = client.ccr.follow(
index="follower_index",
wait_for_active_shards="1",
remote_cluster="remote_cluster",
leader_index="leader_index",
settings={
"index.number_of_replicas": 0
},
max_read_request_operation_count=1024,
max_outstanding_read_requests=16,
max_read_request_size="1024k",
max_write_request_operation_count=32768,
max_write_request_size="16k",
max_outstanding_write_requests=8,
max_write_buffer_count=512,
max_write_buffer_size="512k",
max_retry_delay="10s",
read_poll_timeout="30s",
)const response = await client.ccr.follow({
index: "follower_index",
wait_for_active_shards: 1,
remote_cluster: "remote_cluster",
leader_index: "leader_index",
settings: {
"index.number_of_replicas": 0,
},
max_read_request_operation_count: 1024,
max_outstanding_read_requests: 16,
max_read_request_size: "1024k",
max_write_request_operation_count: 32768,
max_write_request_size: "16k",
max_outstanding_write_requests: 8,
max_write_buffer_count: 512,
max_write_buffer_size: "512k",
max_retry_delay: "10s",
read_poll_timeout: "30s",
});response = client.ccr.follow(
index: "follower_index",
wait_for_active_shards: "1",
body: {
"remote_cluster": "remote_cluster",
"leader_index": "leader_index",
"settings": {
"index.number_of_replicas": 0
},
"max_read_request_operation_count": 1024,
"max_outstanding_read_requests": 16,
"max_read_request_size": "1024k",
"max_write_request_operation_count": 32768,
"max_write_request_size": "16k",
"max_outstanding_write_requests": 8,
"max_write_buffer_count": 512,
"max_write_buffer_size": "512k",
"max_retry_delay": "10s",
"read_poll_timeout": "30s"
}
)$resp = $client->ccr()->follow([
"index" => "follower_index",
"wait_for_active_shards" => "1",
"body" => [
"remote_cluster" => "remote_cluster",
"leader_index" => "leader_index",
"settings" => [
"index.number_of_replicas" => 0,
],
"max_read_request_operation_count" => 1024,
"max_outstanding_read_requests" => 16,
"max_read_request_size" => "1024k",
"max_write_request_operation_count" => 32768,
"max_write_request_size" => "16k",
"max_outstanding_write_requests" => 8,
"max_write_buffer_count" => 512,
"max_write_buffer_size" => "512k",
"max_retry_delay" => "10s",
"read_poll_timeout" => "30s",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"remote_cluster":"remote_cluster","leader_index":"leader_index","settings":{"index.number_of_replicas":0},"max_read_request_operation_count":1024,"max_outstanding_read_requests":16,"max_read_request_size":"1024k","max_write_request_operation_count":32768,"max_write_request_size":"16k","max_outstanding_write_requests":8,"max_write_buffer_count":512,"max_write_buffer_size":"512k","max_retry_delay":"10s","read_poll_timeout":"30s"}' "$ELASTICSEARCH_URL/follower_index/_ccr/follow?wait_for_active_shards=1"client.ccr().follow(f -> f
.index("follower_index")
.leaderIndex("leader_index")
.maxOutstandingReadRequests(16L)
.maxOutstandingWriteRequests(8)
.maxReadRequestOperationCount(1024)
.maxReadRequestSize("1024k")
.maxRetryDelay(m -> m
.time("10s")
)
.maxWriteBufferCount(512)
.maxWriteBufferSize("512k")
.maxWriteRequestOperationCount(32768)
.maxWriteRequestSize("16k")
.readPollTimeout(r -> r
.time("30s")
)
.remoteCluster("remote_cluster")
.settings(s -> s
.otherSettings("index.number_of_replicas", JsonData.fromJson("0"))
)
.waitForActiveShards(w -> w
.count(1)
)
);
{
"remote_cluster" : "remote_cluster",
"leader_index" : "leader_index",
"settings": {
"index.number_of_replicas": 0
},
"max_read_request_operation_count" : 1024,
"max_outstanding_read_requests" : 16,
"max_read_request_size" : "1024k",
"max_write_request_operation_count" : 32768,
"max_write_request_size" : "16k",
"max_outstanding_write_requests" : 8,
"max_write_buffer_count" : 512,
"max_write_buffer_size" : "512k",
"max_retry_delay" : "10s",
"read_poll_timeout" : "30s"
}{
"follow_index_created" : true,
"follow_index_shards_acked" : true,
"index_following_started" : true
}Remove the cross-cluster replication follower retention leases from the leader.
A following index takes out retention leases on its leader index. These leases are used to increase the likelihood that the shards of the leader index retain the history of operations that the shards of the following index need to run replication. When a follower index is converted to a regular index by the unfollow API (either by directly calling the API or by index lifecycle management tasks), these leases are removed. However, removal of the leases can fail, for example when the remote cluster containing the leader index is unavailable. While the leases will eventually expire on their own, their extended existence can cause the leader index to hold more history than necessary and prevent index lifecycle management from performing some operations on the leader index. This API exists to enable manually removing the leases when the unfollow API is unable to do so.
NOTE: This API does not stop replication by a following index. If you use this API with a follower index that is still actively following, the following index will add back retention leases on the leader. The only purpose of this API is to handle the case of failure to remove the following retention leases after the unfollow API is invoked.
the name of the leader index for which specified follower retention leases should be removed
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
POST /<leader_index>/_ccr/forget_follower
{
"follower_cluster" : "<follower_cluster>",
"follower_index" : "<follower_index>",
"follower_index_uuid" : "<follower_index_uuid>",
"leader_remote_cluster" : "<leader_remote_cluster>"
}resp = client.ccr.forget_follower(
index="<leader_index>",
follower_cluster="<follower_cluster>",
follower_index="<follower_index>",
follower_index_uuid="<follower_index_uuid>",
leader_remote_cluster="<leader_remote_cluster>",
)const response = await client.ccr.forgetFollower({
index: "<leader_index>",
follower_cluster: "<follower_cluster>",
follower_index: "<follower_index>",
follower_index_uuid: "<follower_index_uuid>",
leader_remote_cluster: "<leader_remote_cluster>",
});response = client.ccr.forget_follower(
index: "<leader_index>",
body: {
"follower_cluster": "<follower_cluster>",
"follower_index": "<follower_index>",
"follower_index_uuid": "<follower_index_uuid>",
"leader_remote_cluster": "<leader_remote_cluster>"
}
)$resp = $client->ccr()->forgetFollower([
"index" => "<leader_index>",
"body" => [
"follower_cluster" => "<follower_cluster>",
"follower_index" => "<follower_index>",
"follower_index_uuid" => "<follower_index_uuid>",
"leader_remote_cluster" => "<leader_remote_cluster>",
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"follower_cluster":"<follower_cluster>","follower_index":"<follower_index>","follower_index_uuid":"<follower_index_uuid>","leader_remote_cluster":"<leader_remote_cluster>"}' "$ELASTICSEARCH_URL/<leader_index>/_ccr/forget_follower"client.ccr().forgetFollower(f -> f
.followerCluster("<follower_cluster>")
.followerIndex("<follower_index>")
.followerIndexUuid("<follower_index_uuid>")
.index("<leader_index>")
.leaderRemoteCluster("<leader_remote_cluster>")
);
{
"follower_cluster" : "<follower_cluster>",
"follower_index" : "<follower_index>",
"follower_index_uuid" : "<follower_index_uuid>",
"leader_remote_cluster" : "<leader_remote_cluster>"
}{
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0,
"failures" : [ ]
}
}Pause a cross-cluster replication auto-follow pattern. When the API returns, the auto-follow pattern is inactive. New indices that are created on the remote cluster and match the auto-follow patterns are ignored.
You can resume auto-following with the resume auto-follow pattern API. When it resumes, the auto-follow pattern is active again and automatically configures follower indices for newly created indices on the remote cluster that match its patterns. Remote indices that were created while the pattern was paused will also be followed, unless they have been deleted or closed in the interim.
manage_ccrPOST /_ccr/auto_follow/my_auto_follow_pattern/pause
resp = client.ccr.pause_auto_follow_pattern(
name="my_auto_follow_pattern",
)const response = await client.ccr.pauseAutoFollowPattern({
name: "my_auto_follow_pattern",
});response = client.ccr.pause_auto_follow_pattern(
name: "my_auto_follow_pattern"
)$resp = $client->ccr()->pauseAutoFollowPattern([
"name" => "my_auto_follow_pattern",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_ccr/auto_follow/my_auto_follow_pattern/pause"client.ccr().pauseAutoFollowPattern(p -> p
.name("my_auto_follow_pattern")
);
{
"acknowledged" : true
}Convert a cross-cluster replication follower index to a regular index. The API stops the following task associated with a follower index and removes index metadata and settings associated with cross-cluster replication. The follower index must be paused and closed before you call the unfollow API.
Currently cross-cluster replication does not support converting an existing regular index to a follower index. Converting a follower index to a regular index is an irreversible operation.
manage_follow_indexPOST /follower_index/_ccr/unfollow
resp = client.ccr.unfollow(
index="follower_index",
)const response = await client.ccr.unfollow({
index: "follower_index",
});response = client.ccr.unfollow(
index: "follower_index"
)$resp = $client->ccr()->unfollow([
"index" => "follower_index",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/follower_index/_ccr/unfollow"client.ccr().unfollow(u -> u
.index("follower_index")
);
{
"acknowledged" : true
}Comma-separated list of data stream names used to limit the request.
Wildcard (*) expressions are supported. If omitted, all data streams are returned.
Type of data stream that wildcard patterns can match.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, returns all relevant default configurations for the index template.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Whether the maximum timestamp for each data stream should be calculated and returned.
GET _data_stream/my-data-stream
resp = client.indices.get_data_stream(
name="my-data-stream",
)const response = await client.indices.getDataStream({
name: "my-data-stream",
});response = client.indices.get_data_stream(
name: "my-data-stream"
)$resp = $client->indices()->getDataStream([
"name" => "my-data-stream",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/my-data-stream"client.indices().getDataStream(g -> g
.name("my-data-stream")
);
{
"data_streams": [
{
"name": "my-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-2099.03.07-000001",
"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
"prefer_ilm": true,
"ilm_policy": "my-lifecycle-policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-my-data-stream-2099.03.08-000002",
"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
"prefer_ilm": true,
"ilm_policy": "my-lifecycle-policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 2,
"_meta": {
"my-meta-field": "foo"
},
"status": "GREEN",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": true,
"template": "my-index-template",
"ilm_policy": "my-lifecycle-policy",
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"rollover_on_write": false
},
{
"name": "my-data-stream-two",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-two-2099.03.08-000001",
"index_uuid": "3liBu2SYS5axasRt6fUIpA",
"prefer_ilm": true,
"ilm_policy": "my-lifecycle-policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 1,
"_meta": {
"my-meta-field": "foo"
},
"status": "YELLOW",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": true,
"template": "my-index-template",
"ilm_policy": "my-lifecycle-policy",
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"rollover_on_write": false
}
]
}Name of the data stream, which must meet the following criteria:
Lowercase only;
Cannot include \, /, *, ?, ", <, >, |, ,, #, :, or a space character;
Cannot start with -, _, +, or .ds-;
Cannot be . or ..;
Cannot be longer than 255 bytes. Multi-byte characters count towards this limit faster.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
PUT _data_stream/logs-foo-bar
resp = client.indices.create_data_stream(
name="logs-foo-bar",
)const response = await client.indices.createDataStream({
name: "logs-foo-bar",
});response = client.indices.create_data_stream(
name: "logs-foo-bar"
)$resp = $client->indices()->createDataStream([
"name" => "logs-foo-bar",
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/logs-foo-bar"client.indices().createDataStream(c -> c
.name("logs-foo-bar")
);
Comma-separated list of data streams to delete. Wildcard (*) expressions are supported.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Type of data stream that wildcard patterns can match. Supports comma-separated values,such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
DELETE _data_stream/my-data-stream
resp = client.indices.delete_data_stream(
name="my-data-stream",
)const response = await client.indices.deleteDataStream({
name: "my-data-stream",
});response = client.indices.delete_data_stream(
name: "my-data-stream"
)$resp = $client->indices()->deleteDataStream([
"name" => "my-data-stream",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/my-data-stream"client.indices().deleteDataStream(d -> d
.name("my-data-stream")
);
Comma-separated list of data streams used to limit the request.
Wildcard expressions (*) are supported.
To target all data streams in a cluster, omit this parameter or use *.
Type of data stream that wildcard patterns can match.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
GET /_data_stream/my-index-000001/_stats
resp = client.indices.data_streams_stats(
name="my-index-000001",
)const response = await client.indices.dataStreamsStats({
name: "my-index-000001",
});response = client.indices.data_streams_stats(
name: "my-index-000001"
)$resp = $client->indices()->dataStreamsStats([
"name" => "my-index-000001",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/my-index-000001/_stats"client.indices().dataStreamsStats(d -> d
.name("my-index-000001")
);
{
"_shards": {
"total": 10,
"successful": 5,
"failed": 0
},
"data_stream_count": 2,
"backing_indices": 5,
"total_store_size": "7kb",
"total_store_size_bytes": 7268,
"data_streams": [
{
"data_stream": "my-data-stream",
"backing_indices": 3,
"store_size": "3.7kb",
"store_size_bytes": 3772,
"maximum_timestamp": 1607512028000
},
{
"data_stream": "my-data-stream-two",
"backing_indices": 2,
"store_size": "3.4kb",
"store_size_bytes": 3496,
"maximum_timestamp": 1607425567000
}
]
}Aggregate a time series (TSDS) index and store pre-computed statistical summaries (min, max, sum, value_count and avg) for each metric field grouped by a configured time interval.
For example, a TSDS index that contains metrics sampled every 10 seconds can be downsampled to an hourly index.
All documents within an hour interval are summarized and stored as a single document in the downsample index.
NOTE: Only indices in a time series data stream are supported.
Neither field nor document level security can be defined on the source index.
The source index must be read only (index.blocks.write: true).
POST /my-time-series-index/_downsample/my-downsampled-time-series-index
{
"fixed_interval": "1d"
}resp = client.indices.downsample(
index="my-time-series-index",
target_index="my-downsampled-time-series-index",
config={
"fixed_interval": "1d"
},
)const response = await client.indices.downsample({
index: "my-time-series-index",
target_index: "my-downsampled-time-series-index",
config: {
fixed_interval: "1d",
},
});response = client.indices.downsample(
index: "my-time-series-index",
target_index: "my-downsampled-time-series-index",
body: {
"fixed_interval": "1d"
}
)$resp = $client->indices()->downsample([
"index" => "my-time-series-index",
"target_index" => "my-downsampled-time-series-index",
"body" => [
"fixed_interval" => "1d",
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"fixed_interval":"1d"}' "$ELASTICSEARCH_URL/my-time-series-index/_downsample/my-downsampled-time-series-index"client.indices().downsample(d -> d
.index("my-time-series-index")
.targetIndex("my-downsampled-time-series-index")
.config(c -> c
.fixedInterval(f -> f
.time("1d")
)
)
);
{
"fixed_interval": "1d"
}Get information about an index or data stream's current data stream lifecycle status, such as time since index creation, time since rollover, the lifecycle configuration managing the index, or any errors encountered during lifecycle execution.
GET .ds-metrics-2023.03.22-000001/_lifecycle/explain
resp = client.indices.explain_data_lifecycle(
index=".ds-metrics-2023.03.22-000001",
)const response = await client.indices.explainDataLifecycle({
index: ".ds-metrics-2023.03.22-000001",
});response = client.indices.explain_data_lifecycle(
index: ".ds-metrics-2023.03.22-000001"
)$resp = $client->indices()->explainDataLifecycle([
"index" => ".ds-metrics-2023.03.22-000001",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/.ds-metrics-2023.03.22-000001/_lifecycle/explain"client.indices().explainDataLifecycle(e -> e
.index(".ds-metrics-2023.03.22-000001")
);
{
"indices": {
".ds-metrics-2023.03.22-000001": {
"index" : ".ds-metrics-2023.03.22-000001",
"managed_by_lifecycle" : true,
"index_creation_date_millis" : 1679475563571,
"time_since_index_creation" : "843ms",
"rollover_date_millis" : 1679475564293,
"time_since_rollover" : "121ms",
"lifecycle" : { },
"generation_time" : "121ms"
}
}{
"indices": {
".ds-metrics-2023.03.22-000001": {
"index" : ".ds-metrics-2023.03.22-000001",
"managed_by_lifecycle" : true,
"index_creation_date_millis" : 1679475563571,
"time_since_index_creation" : "843ms",
"lifecycle" : {
"enabled": true
},
"error": "{\"type\":\"validation_exception\",\"reason\":\"Validation Failed: 1: this action would add [2] shards, but this cluster
currently has [4]/[3] maximum normal shards open;\"}"
}
}Converts an index alias to a data stream.
You must have a matching index template that is data stream enabled.
The alias must meet the following criteria:
The alias must have a write index;
All indices for the alias must have a @timestamp field mapping of a date or date_nanos field type;
The alias must not have any filters;
The alias must not use custom routing.
If successful, the request removes the alias and creates a data stream with the same name.
The indices for the alias become hidden backing indices for the stream.
The write index for the alias becomes the write index for the stream.
managePeriod to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
POST _data_stream/_migrate/my-time-series-data
resp = client.indices.migrate_to_data_stream(
name="my-time-series-data",
)const response = await client.indices.migrateToDataStream({
name: "my-time-series-data",
});response = client.indices.migrate_to_data_stream(
name: "my-time-series-data"
)$resp = $client->indices()->migrateToDataStream([
"name" => "my-time-series-data",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/_migrate/my-time-series-data"client.indices().migrateToDataStream(m -> m
.name("my-time-series-data")
);
Performs one or more data stream modification actions in a single atomic operation.
POST _data_stream/_modify
{
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}resp = client.indices.modify_data_stream(
actions=[
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
],
)const response = await client.indices.modifyDataStream({
actions: [
{
remove_backing_index: {
data_stream: "my-data-stream",
index: ".ds-my-data-stream-2023.07.26-000001",
},
},
{
add_backing_index: {
data_stream: "my-data-stream",
index: ".ds-my-data-stream-2023.07.26-000001-downsample",
},
},
],
});response = client.indices.modify_data_stream(
body: {
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}
)$resp = $client->indices()->modifyDataStream([
"body" => [
"actions" => array(
[
"remove_backing_index" => [
"data_stream" => "my-data-stream",
"index" => ".ds-my-data-stream-2023.07.26-000001",
],
],
[
"add_backing_index" => [
"data_stream" => "my-data-stream",
"index" => ".ds-my-data-stream-2023.07.26-000001-downsample",
],
],
),
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"actions":[{"remove_backing_index":{"data_stream":"my-data-stream","index":".ds-my-data-stream-2023.07.26-000001"}},{"add_backing_index":{"data_stream":"my-data-stream","index":".ds-my-data-stream-2023.07.26-000001-downsample"}}]}' "$ELASTICSEARCH_URL/_data_stream/_modify"client.indices().modifyDataStream(m -> m
.actions(List.of(Action.of(a -> a
.removeBackingIndex(r -> r
.dataStream("my-data-stream")
.index(".ds-my-data-stream-2023.07.26-000001")
)),Action.of(ac -> ac
.addBackingIndex(ad -> ad
.dataStream("my-data-stream")
.index(".ds-my-data-stream-2023.07.26-000001-downsample")
))))
);
{
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}Promote a data stream from a replicated data stream managed by cross-cluster replication (CCR) to a regular data stream.
With CCR auto following, a data stream from a remote cluster can be replicated to the local cluster. These data streams can't be rolled over in the local cluster. These replicated data streams roll over only if the upstream data stream rolls over. In the event that the remote cluster is no longer available, the data stream in the local cluster can be promoted to a regular data stream, which allows these data streams to be rolled over in the local cluster.
NOTE: When promoting a data stream, ensure the local cluster has a data stream enabled index template that matches the data stream. If this is missing, the data stream will not be able to roll over until a matching index template is created. This will affect the lifecycle management of the data stream and interfere with the data stream size and retention.
POST /_data_stream/_promote/my-data-stream
resp = client.indices.promote_data_stream(
name="my-data-stream",
)const response = await client.indices.promoteDataStream({
name: "my-data-stream",
});response = client.indices.promote_data_stream(
name: "my-data-stream"
)$resp = $client->indices()->promoteDataStream([
"name" => "my-data-stream",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/_promote/my-data-stream"client.indices().promoteDataStream(p -> p
.name("my-data-stream")
);
All methods and paths for this operation:
Perform multiple index, create, delete, and update actions in a single request.
This reduces overhead and can greatly increase indexing speed.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:
create action, you must have the create_doc, create, index, or write index privilege. Data streams support only the create action.index action, you must have the create, index, or write index privilege.delete action, you must have the delete or write index privilege.update action, you must have the index or write index privilege.auto_configure, create_index, or manage index privilege.refresh parameter, you must have the maintenance or manage index privilege.Automatic data stream creation requires a matching index template with data stream enabled.
The actions are specified in the request body using a newline delimited JSON (NDJSON) structure:
action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n
The index and create actions expect a source on the next line and have the same semantics as the op_type parameter in the standard index API.
A create action fails if a document with the same ID already exists in the target
An index action adds or replaces a document as necessary.
NOTE: Data streams support only the create action.
To update or delete a document in a data stream, you must target the backing index containing the document.
An update action expects that the partial doc, upsert, and script and its options are specified on the next line.
A delete action does not expect a source on the next line and has the same semantics as the standard delete API.
NOTE: The final line of data must end with a newline character (\n).
Each newline character may be preceded by a carriage return (\r).
When sending NDJSON data to the _bulk endpoint, use a Content-Type header of application/json or application/x-ndjson.
Because this format uses literal newline characters (\n) as delimiters, make sure that the JSON actions and sources are not pretty printed.
If you provide a target in the request path, it is used for any actions that don't explicitly specify an _index argument.
A note on the format: the idea here is to make processing as fast as possible.
As some of the actions are redirected to other shards on other nodes, only action_meta_data is parsed on the receiving node side.
Client libraries using this protocol should try and strive to do something similar on the client side, and reduce buffering as much as possible.
There is no "correct" number of actions to perform in a single bulk request. Experiment with different settings to find the optimal size for your particular workload. Note that Elasticsearch limits the maximum size of a HTTP request to 100mb by default so clients must ensure that no request exceeds this size. It is not possible to index a single document that exceeds the size limit, so you must pre-process any such documents into smaller pieces before sending them to Elasticsearch. For instance, split documents into pages or chapters before indexing them, or store raw binary data in a system outside Elasticsearch and replace the raw data with a link to the external system in the documents that you send to Elasticsearch.
Client suppport for bulk requests
Some of the officially supported clients provide helpers to assist with bulk requests and reindexing:
esutil.BulkIndexerSearch::Elasticsearch::Client::5_0::Bulk and Search::Elasticsearch::Client::5_0::Scrollelasticsearch.helpers.*client.helpers.*BulkAllObservableElasticsearch::Helpers::BulkHelperSubmitting bulk requests with cURL
If you're providing text file input to curl, you must use the --data-binary flag instead of plain -d.
The latter doesn't preserve newlines. For example:
$ cat requests
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
$ curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/_bulk --data-binary "@requests"; echo
{"took":7, "errors": false, "items":[{"index":{"_index":"test","_id":"1","_version":1,"result":"created","forced_refresh":false}}]}
Optimistic concurrency control
Each index and delete action within a bulk API call may include the if_seq_no and if_primary_term parameters in their respective action and meta data lines.
The if_seq_no and if_primary_term parameters control how operations are run, based on the last modification to existing documents. See Optimistic concurrency control for more details.
Versioning
Each bulk item can include the version value using the version field.
It automatically follows the behavior of the index or delete operation based on the _version mapping.
It also support the version_type.
Routing
Each bulk item can include the routing value using the routing field.
It automatically follows the behavior of the index or delete operation based on the _routing mapping.
NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing setting enabled in the template.
Wait for active shards
When making bulk calls, you can set the wait_for_active_shards parameter to require a minimum number of shard copies to be active before starting to process the bulk request.
Refresh
Control when the changes made by this request are visible to search.
NOTE: Only the shards that receive the bulk request will be affected by refresh.
Imagine a _bulk?refresh=wait_for request with three documents in it that happen to be routed to different shards in an index with five shards.
The request will only wait for those three shards to refresh.
The other two shards that make up the index do not participate in the _bulk request at all.
True or false if to include the document source in the error message in case of parsing errors.
If true, the response will include the ingest pipelines that were run for each index or create.
The pipeline identifier to use to preprocess incoming documents.
If the index has a default ingest pipeline specified, setting the value to _none turns off the default ingest pipeline for this request.
If a final pipeline is configured, it will always run regardless of the value of this parameter.
If true, Elasticsearch refreshes the affected shards to make this operation visible to search.
If wait_for, wait for a refresh to make this operation visible to search.
If false, do nothing with refreshes.
Valid values: true, false, wait_for.
Values are true, false, or wait_for.
A custom value that is used to route operations to a specific shard.
Indicates whether to return the _source field (true or false) or contains a list of fields to return.
A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in _source_includes query parameter.
If the _source parameter is false, this parameter is ignored.
A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned.
You can exclude fields from this subset using the _source_excludes query parameter.
If the _source parameter is false, this parameter is ignored.
The period each action waits for the following operations: automatic index creation, dynamic mapping updates, and waiting for active shards.
The default is 1m (one minute), which guarantees Elasticsearch waits for at least the timeout before failing.
The actual wait time could be longer, particularly when multiple waits occur.
Values are -1 or 0.
The number of shard copies that must be active before proceeding with the operation.
Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
The default is 1, which waits for each primary shard to be active.
Values are all or index-setting.
If true, the request's actions must target an index alias.
If true, the request's actions must target a data stream (existing or to be created).
POST _bulk
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }resp = client.bulk(
operations=[
{
"index": {
"_index": "test",
"_id": "1"
}
},
{
"field1": "value1"
},
{
"delete": {
"_index": "test",
"_id": "2"
}
},
{
"create": {
"_index": "test",
"_id": "3"
}
},
{
"field1": "value3"
},
{
"update": {
"_id": "1",
"_index": "test"
}
},
{
"doc": {
"field2": "value2"
}
}
],
)const response = await client.bulk({
operations: [
{
index: {
_index: "test",
_id: "1",
},
},
{
field1: "value1",
},
{
delete: {
_index: "test",
_id: "2",
},
},
{
create: {
_index: "test",
_id: "3",
},
},
{
field1: "value3",
},
{
update: {
_id: "1",
_index: "test",
},
},
{
doc: {
field2: "value2",
},
},
],
});response = client.bulk(
body: [
{
"index": {
"_index": "test",
"_id": "1"
}
},
{
"field1": "value1"
},
{
"delete": {
"_index": "test",
"_id": "2"
}
},
{
"create": {
"_index": "test",
"_id": "3"
}
},
{
"field1": "value3"
},
{
"update": {
"_id": "1",
"_index": "test"
}
},
{
"doc": {
"field2": "value2"
}
}
]
)$resp = $client->bulk([
"body" => array(
[
"index" => [
"_index" => "test",
"_id" => "1",
],
],
[
"field1" => "value1",
],
[
"delete" => [
"_index" => "test",
"_id" => "2",
],
],
[
"create" => [
"_index" => "test",
"_id" => "3",
],
],
[
"field1" => "value3",
],
[
"update" => [
"_id" => "1",
"_index" => "test",
],
],
[
"doc" => [
"field2" => "value2",
],
],
),
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '[{"index":{"_index":"test","_id":"1"}},{"field1":"value1"},{"delete":{"_index":"test","_id":"2"}},{"create":{"_index":"test","_id":"3"}},{"field1":"value3"},{"update":{"_id":"1","_index":"test"}},{"doc":{"field2":"value2"}}]' "$ELASTICSEARCH_URL/_bulk"{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_id" : "2" } }
{ "create" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }{ "update" : {"_id" : "1", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"} }
{ "update" : { "_id" : "0", "_index" : "index1", "retry_on_conflict" : 3} }
{ "script" : { "source": "ctx._source.counter += params.param1", "lang" : "painless", "params" : {"param1" : 1}}, "upsert" : {"counter" : 1}}
{ "update" : {"_id" : "2", "_index" : "index1", "retry_on_conflict" : 3} }
{ "doc" : {"field" : "value"}, "doc_as_upsert" : true }
{ "update" : {"_id" : "3", "_index" : "index1", "_source" : true} }
{ "doc" : {"field" : "value"} }
{ "update" : {"_id" : "4", "_index" : "index1"} }
{ "doc" : {"field" : "value"}, "_source": true}{ "update": {"_id": "5", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "update": {"_id": "6", "_index": "index1"} }
{ "doc": {"my_field": "foo"} }
{ "create": {"_id": "7", "_index": "index1"} }
{ "my_field": "foo" }{ "index" : { "_index" : "my_index", "_id" : "1", "dynamic_templates": {"work_location": "geo_point"}} }
{ "field" : "value1", "work_location": "41.12,-71.34", "raw_location": "41.12,-71.34"}
{ "create" : { "_index" : "my_index", "_id" : "2", "dynamic_templates": {"home_location": "geo_point"}} }
{ "field" : "value2", "home_location": "41.12,-71.34"}{
"took": 30,
"errors": false,
"items": [
{
"index": {
"_index": "test",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 0,
"_primary_term": 1
}
},
{
"delete": {
"_index": "test",
"_id": "2",
"_version": 1,
"result": "not_found",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 404,
"_seq_no" : 1,
"_primary_term" : 2
}
},
{
"create": {
"_index": "test",
"_id": "3",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 201,
"_seq_no" : 2,
"_primary_term" : 3
}
},
{
"update": {
"_index": "test",
"_id": "1",
"_version": 2,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"status": 200,
"_seq_no" : 3,
"_primary_term" : 4
}
}
]
}{
"took": 486,
"errors": true,
"items": [
{
"update": {
"_index": "index1",
"_id": "5",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"_index": "index1",
"_id": "6",
"status": 404,
"error": {
"type": "document_missing_exception",
"reason": "[6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"create": {
"_index": "index1",
"_id": "7",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1,
"status": 201
}
}
]
}{
"items": [
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[5]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
},
{
"update": {
"error": {
"type": "document_missing_exception",
"reason": "[6]: document missing",
"index_uuid": "aAsFqTI0Tc2W0LCWgPNrOA",
"shard": "0",
"index": "index1"
}
}
}
]
}Get a document and its source or stored fields from an index.
By default, this API is realtime and is not affected by the refresh rate of the index (when data will become visible for search).
In the case where stored fields are requested with the stored_fields parameter and the document has been updated but is not yet refreshed, the API will have to parse and analyze the source to extract the stored fields.
To turn off realtime behavior, set the realtime parameter to false.
Source filtering
By default, the API returns the contents of the _source field unless you have used the stored_fields parameter or the _source field is turned off.
You can turn off _source retrieval by using the _source parameter:
GET my-index-000001/_doc/0?_source=false
If you only need one or two fields from the _source, use the _source_includes or _source_excludes parameters to include or filter out particular fields.
This can be helpful with large documents where partial retrieval can save on network overhead
Both parameters take a comma separated list of fields or wildcard expressions.
For example:
GET my-index-000001/_doc/0?_source_includes=*.id&_source_excludes=entities
If you only want to specify includes, you can use a shorter notation:
GET my-index-000001/_doc/0?_source=*.id
Routing
If routing is used during indexing, the routing value also needs to be specified to retrieve a document. For example:
GET my-index-000001/_doc/2?routing=user1
This request gets the document with ID 2, but it is routed based on the user. The document is not fetched if the correct routing is not specified.
Distributed
The GET operation is hashed into a specific shard ID. It is then redirected to one of the replicas within that shard ID and returns the result. The replicas are the primary shard and its replicas within that shard ID group. This means that the more replicas you have, the better your GET scaling will be.
Versioning support
You can use the version parameter to retrieve the document only if its current version is equal to the specified one.
Internally, Elasticsearch has marked the old document as deleted and added an entirely new document. The old version of the document doesn't disappear immediately, although you won't be able to access it. Elasticsearch cleans up deleted documents in the background as you continue to index more data.
readThe node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.
If it is set to _local, the operation will prefer to be run on a local allocated shard when possible.
If it is set to a custom value, the value is used to guarantee that the same shards will be used for the same custom value.
This can help with "jumping values" when hitting different shards in different refresh states.
A sample value can be something like the web session ID or the user name.
If true, the request is real-time as opposed to near-real-time.
If true, the request refreshes the relevant shards before retrieving the document.
Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
A custom value used to route operations to a specific shard.
Indicates whether to return the _source field (true or false) or lists the fields to return.
A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in _source_includes query parameter.
If the _source parameter is false, this parameter is ignored.
A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned.
You can exclude fields from this subset using the _source_excludes query parameter.
If the _source parameter is false, this parameter is ignored.
A comma-separated list of stored fields to return as part of a hit.
If no fields are specified, no stored fields are included in the response.
If this field is specified, the _source parameter defaults to false.
Only leaf fields can be retrieved with the stored_field option.
Object fields can't be returned;if specified, the request fails.
The version number for concurrency control. It must match the current version of the document for the request to succeed.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
GET my-index-000001/_doc/1?stored_fields=tags,counter
resp = client.get(
index="my-index-000001",
id="1",
stored_fields="tags,counter",
)const response = await client.get({
index: "my-index-000001",
id: 1,
stored_fields: "tags,counter",
});response = client.get(
index: "my-index-000001",
id: "1",
stored_fields: "tags,counter"
)$resp = $client->get([
"index" => "my-index-000001",
"id" => "1",
"stored_fields" => "tags,counter",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_doc/1?stored_fields=tags,counter"{
"_index": "my-index-000001",
"_id": "0",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"found": true,
"_source": {
"@timestamp": "2099-11-15T14:12:12",
"http": {
"request": {
"method": "get"
},
"response": {
"status_code": 200,
"bytes": 1070000
},
"version": "1.1"
},
"source": {
"ip": "127.0.0.1"
},
"message": "GET /search HTTP/1.1 200 1070000",
"user": {
"id": "kimchy"
}
}
}{
"_index": "my-index-000001",
"_id": "1",
"_version": 1,
"_seq_no" : 22,
"_primary_term" : 1,
"found": true,
"fields": {
"tags": [
"production"
]
}
}{
"_index": "my-index-000001",
"_id": "2",
"_version": 1,
"_seq_no" : 13,
"_primary_term" : 1,
"_routing": "user1",
"found": true,
"fields": {
"tags": [
"env2"
]
}
}Remove a JSON document from the specified index.
NOTE: You cannot send deletion requests directly to a data stream. To delete a document in a data stream, you must target the backing index containing the document.
Optimistic concurrency control
Delete operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters.
If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409.
Versioning
Each document indexed is versioned.
When deleting a document, the version can be specified to make sure the relevant document you are trying to delete is actually being deleted and it has not changed in the meantime.
Every write operation run on a document, deletes included, causes its version to be incremented.
The version number of a deleted document remains available for a short time after deletion to allow for control of concurrent operations.
The length of time for which a deleted document's version remains available is determined by the index.gc_deletes index setting.
Routing
If routing is used during indexing, the routing value also needs to be specified to delete a document.
If the _routing mapping is set to required and no routing value is specified, the delete API throws a RoutingMissingException and rejects the request.
For example:
DELETE /my-index-000001/_doc/1?routing=shard-1
This request deletes the document with ID 1, but it is routed based on the user. The document is not deleted if the correct routing is not specified.
Distributed
The delete operation gets hashed into a specific shard ID. It then gets redirected into the primary shard within that ID group and replicated (if needed) to shard replicas within that ID group.
deleteOnly perform the operation if the document has this primary term.
Only perform the operation if the document has this sequence number.
If true, Elasticsearch refreshes the affected shards to make this operation visible to search.
If wait_for, it waits for a refresh to make this operation visible to search.
If false, it does nothing with refreshes.
Values are true, false, or wait_for.
A custom value used to route operations to a specific shard.
The period to wait for active shards.
This parameter is useful for situations where the primary shard assigned to perform the delete operation might not be available when the delete operation runs. Some reasons for this might be that the primary shard is currently recovering from a store or undergoing relocation. By default, the delete operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error.
Values are -1 or 0.
An explicit version number for concurrency control. It must match the current version of the document for the request to succeed.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
The minimum number of shard copies that must be active before proceeding with the operation.
You can set it to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
The default value of 1 means it waits for each primary shard to be active.
Values are all or index-setting.
DELETE /my-index-000001/_doc/1
resp = client.delete(
index="my-index-000001",
id="1",
)const response = await client.delete({
index: "my-index-000001",
id: 1,
});response = client.delete(
index: "my-index-000001",
id: "1"
)$resp = $client->delete([
"index" => "my-index-000001",
"id" => "1",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_doc/1"client.delete(d -> d
.id("1")
.index("my-index-000001")
);
{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
},
"_index": "my-index-000001",
"_id": "1",
"_version": 2,
"_primary_term": 1,
"_seq_no": 5,
"result": "deleted"
}Deletes documents that match the specified query.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or alias:
readdelete or writeYou can specify the query criteria in the request URI or the request body using the same syntax as the search API. When you submit a delete by query request, Elasticsearch gets a snapshot of the data stream or index when it begins processing the request and deletes matching documents using internal versioning. If a document changes between the time that the snapshot is taken and the delete operation is processed, it results in a version conflict and the delete operation fails.
NOTE: Documents with a version equal to 0 cannot be deleted using delete by query because internal versioning does not support 0 as a valid version number.
While processing a delete by query request, Elasticsearch performs multiple search requests sequentially to find all of the matching documents to delete. A bulk delete request is performed for each batch of matching documents. If a search or bulk request is rejected, the requests are retried up to 10 times, with exponential back off. If the maximum retry limit is reached, processing halts and all failed requests are returned in the response. Any delete requests that completed successfully still stick, they are not rolled back.
You can opt to count version conflicts instead of halting and returning by setting conflicts to proceed.
Note that if you opt to count version conflicts the operation could attempt to delete more documents from the source than max_docs until it has successfully deleted max_docs documents, or it has gone through every document in the source query.
Throttling delete requests
To control the rate at which delete by query issues batches of delete operations, you can set requests_per_second to any positive decimal number.
This pads each batch with a wait time to throttle the rate.
Set requests_per_second to -1 to disable throttling.
Throttling uses a wait time between batches so that the internal scroll requests can be given a timeout that takes the request padding into account.
The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing.
By default the batch size is 1000, so if requests_per_second is set to 500:
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
Since the batch is issued as a single _bulk request, large batch sizes cause Elasticsearch to create many requests and wait before starting the next set.
This is "bursty" instead of "smooth".
Slicing
Delete by query supports sliced scroll to parallelize the delete process. This can improve efficiency and provide a convenient way to break the request down into smaller parts.
Setting slices to auto lets Elasticsearch choose the number of slices to use.
This setting will use one slice per shard, up to a certain limit.
If there are multiple source data streams or indices, it will choose the number of slices based on the index or backing index with the smallest number of shards.
Adding slices to the delete by query operation creates sub-requests which means it has some quirks:
slices will rethrottle the unfinished sub-request proportionally.slices will cancel each sub-request.slices each sub-request won't get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.requests_per_second and max_docs on a request with slices are distributed proportionally to each sub-request. Combine that with the earlier point about distribution being uneven and you should conclude that using max_docs with slices might not result in exactly max_docs documents being deleted.If you're slicing manually or otherwise tuning automatic slicing, keep in mind that:
slices hurts performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.Whether query or delete performance dominates the runtime depends on the documents being reindexed and cluster resources.
Cancel a delete by query operation
Any delete by query can be canceled using the task cancel API. For example:
POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel
The task ID can be found by using the get tasks API.
Cancellation should happen quickly but might take a few seconds. The get task status API will continue to list the delete by query task until this task checks that it has been cancelled and terminates itself.
read,deleteA comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (*).
To search all data streams or indices, omit this parameter or use * or _all.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.
Analyzer to use for the query string.
This parameter can be used only when the q query string parameter is specified.
If true, wildcard and prefix queries are analyzed.
This parameter can be used only when the q query string parameter is specified.
What to do if delete by query hits version conflicts: abort or proceed.
Supported values include:
abort: Stop reindexing if there are conflicts.proceed: Continue reindexing even if there are conflicts.Values are abort or proceed.
The default operator for query string query: AND or OR.
This parameter can be used only when the q query string parameter is specified.
Values are and, AND, or, or OR.
The field to use as default where no field prefix is given in the query string.
This parameter can be used only when the q query string parameter is specified.
The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
It supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Skips the specified number of documents.
If true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
This parameter can be used only when the q query string parameter is specified.
The maximum number of documents to process.
Defaults to all documents.
When set to a value less then or equal to scroll_size, a scroll will not be used to retrieve the results for the operation.
The node or shard the operation should be performed on. It is random by default.
If true, Elasticsearch refreshes all shards involved in the delete by query after the request completes.
This is different than the delete API's refresh parameter, which causes just the shard that received the delete request to be refreshed.
Unlike the delete API, it does not support wait_for.
If true, the request cache is used for this request.
Defaults to the index-level setting.
The throttle for this request in sub-requests per second.
A custom value used to route operations to a specific shard.
A query in the Lucene query string syntax.
The period to retain the search context for scrolling.
Values are -1 or 0.
The size of the scroll request that powers the operation.
The explicit timeout for each search request. It defaults to no timeout.
Values are -1 or 0.
The type of the search operation.
Available options include query_then_fetch and dfs_query_then_fetch.
Supported values include:
query_then_fetch: Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.dfs_query_then_fetch: Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.Values are query_then_fetch or dfs_query_then_fetch.
The number of slices this task should be divided into.
Value is auto.
A comma-separated list of <field>:<direction> pairs.
The specific tag of the request for logging and statistical purposes.
The maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.
Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.
The period each deletion request waits for active shards.
Values are -1 or 0.
If true, returns the document version as part of a hit.
The number of shard copies that must be active before proceeding with the operation.
Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
The timeout value controls how long each write request waits for unavailable shards to become available.
Values are all or index-setting.
If true, the request blocks until the operation is complete.
If false, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task. Elasticsearch creates a record of this task as a document at .tasks/task/${taskId}. When you are done with a task, you should delete the task document so Elasticsearch can reclaim the space.
The maximum number of documents to delete.
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
POST /my-index-000001,my-index-000002/_delete_by_query
{
"query": {
"match_all": {}
}
}resp = client.delete_by_query(
index="my-index-000001,my-index-000002",
query={
"match_all": {}
},
)const response = await client.deleteByQuery({
index: "my-index-000001,my-index-000002",
query: {
match_all: {},
},
});response = client.delete_by_query(
index: "my-index-000001,my-index-000002",
body: {
"query": {
"match_all": {}
}
}
)$resp = $client->deleteByQuery([
"index" => "my-index-000001,my-index-000002",
"body" => [
"query" => [
"match_all" => new ArrayObject([]),
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"query":{"match_all":{}}}' "$ELASTICSEARCH_URL/my-index-000001,my-index-000002/_delete_by_query"client.deleteByQuery(d -> d
.index(List.of("my-index-000001","my-index-000002"))
.query(q -> q
.matchAll(m -> m)
)
);
{
"query": {
"match_all": {}
}
}{
"query": {
"term": {
"user.id": "kimchy"
}
},
"max_docs": 1
}{
"slice": {
"id": 0,
"max": 2
},
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}{
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}{
"took" : 147,
"timed_out": false,
"total": 119,
"deleted": 119,
"batches": 1,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"failures" : [ ]
}Change the number of requests per second for a particular delete by query operation. Rethrottling that speeds up the query takes effect immediately but rethrotting that slows down the query takes effect after completing the current batch to prevent scroll timeouts.
POST _delete_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
resp = client.delete_by_query_rethrottle(
task_id="r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second="-1",
)const response = await client.deleteByQueryRethrottle({
task_id: "r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second: "-1",
});response = client.delete_by_query_rethrottle(
task_id: "r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second: "-1"
)$resp = $client->deleteByQueryRethrottle([
"task_id" => "r1A2WoRbTwKZ516z6NEs5A:36619",
"requests_per_second" => "-1",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_delete_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1"client.deleteByQueryRethrottle(d -> d
.requestsPerSecond(-1.0F)
.taskId("r1A2WoRbTwKZ516z6NEs5A:36619")
);
Get the source of a document. For example:
GET my-index-000001/_source/1
You can use the source filtering parameters to control which parts of the _source are returned:
GET my-index-000001/_source/1/?_source_includes=*.id&_source_excludes=entities
readThe node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.
If true, the request is real-time as opposed to near-real-time.
If true, the request refreshes the relevant shards before retrieving the document.
Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
A custom value used to route operations to a specific shard.
Indicates whether to return the _source field (true or false) or lists the fields to return.
A comma-separated list of source fields to exclude in the response.
A comma-separated list of source fields to include in the response.
The version number for concurrency control. It must match the current version of the document for the request to succeed.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
GET my-index-000001/_source/1
resp = client.get_source(
index="my-index-000001",
id="1",
)const response = await client.getSource({
index: "my-index-000001",
id: 1,
});response = client.get_source(
index: "my-index-000001",
id: "1"
)$resp = $client->getSource([
"index" => "my-index-000001",
"id" => "1",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_source/1"client.getSource(g -> g
.id("1")
.index("my-index-000001")
);
Check whether a document source exists in an index. For example:
HEAD my-index-000001/_source/1
A document's source is not available if it is disabled in the mapping.
readA comma-separated list of data streams, indices, and aliases.
It supports wildcards (*).
A unique identifier for the document.
The node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.
If true, the request is real-time as opposed to near-real-time.
If true, the request refreshes the relevant shards before retrieving the document.
Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
A custom value used to route operations to a specific shard.
Indicates whether to return the _source field (true or false) or lists the fields to return.
A comma-separated list of source fields to exclude in the response.
A comma-separated list of source fields to include in the response.
The version number for concurrency control. It must match the current version of the document for the request to succeed.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
HEAD my-index-000001/_source/1
resp = client.exists_source(
index="my-index-000001",
id="1",
)const response = await client.existsSource({
index: "my-index-000001",
id: 1,
});response = client.exists_source(
index: "my-index-000001",
id: "1"
)$resp = $client->existsSource([
"index" => "my-index-000001",
"id" => "1",
]);curl --head -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_source/1"client.existsSource(e -> e
.id("1")
.index("my-index-000001")
);
All methods and paths for this operation:
Get multiple term vectors with a single request.
You can specify existing documents by index and ID or provide artificial documents in the body of the request.
You can specify the index in the request body or request URI.
The response contains a docs array with all the fetched termvectors.
Each element has the structure provided by the termvectors API.
Artificial documents
You can also use mtermvectors to generate term vectors for artificial documents provided in the body of the request.
The mapping used is determined by the specified _index.
readA comma-separated list of documents ids. You must define ids as parameter or set "ids" or "docs" in the request body
A comma-separated list or wildcard expressions of fields to include in the statistics.
It is used as the default list unless a specific field list is provided in the completion_fields or fielddata_fields parameters.
If true, the response includes the document count, sum of document frequencies, and sum of total term frequencies.
If true, the response includes term offsets.
If true, the response includes term payloads.
If true, the response includes term positions.
The node or shard the operation should be performed on. It is random by default.
If true, the request is real-time as opposed to near-real-time.
A custom value used to route operations to a specific shard.
If true, the response includes term frequency and document frequency.
If true, returns the document version as part of a hit.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
POST /my-index-000001/_mtermvectors
{
"docs": [
{
"_id": "2",
"fields": [
"message"
],
"term_statistics": true
},
{
"_id": "1"
}
]
}resp = client.mtermvectors(
index="my-index-000001",
docs=[
{
"_id": "2",
"fields": [
"message"
],
"term_statistics": True
},
{
"_id": "1"
}
],
)const response = await client.mtermvectors({
index: "my-index-000001",
docs: [
{
_id: "2",
fields: ["message"],
term_statistics: true,
},
{
_id: "1",
},
],
});response = client.mtermvectors(
index: "my-index-000001",
body: {
"docs": [
{
"_id": "2",
"fields": [
"message"
],
"term_statistics": true
},
{
"_id": "1"
}
]
}
)$resp = $client->mtermvectors([
"index" => "my-index-000001",
"body" => [
"docs" => array(
[
"_id" => "2",
"fields" => array(
"message",
),
"term_statistics" => true,
],
[
"_id" => "1",
],
),
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"docs":[{"_id":"2","fields":["message"],"term_statistics":true},{"_id":"1"}]}' "$ELASTICSEARCH_URL/my-index-000001/_mtermvectors"client.mtermvectors(m -> m
.docs(List.of(MultiTermVectorsOperation.of(mu -> mu
.id("2")
.fields("message")
.termStatistics(true)),MultiTermVectorsOperation.of(mu -> mu
.id("1"))))
.index("my-index-000001")
);
{
"docs": [
{
"_id": "2",
"fields": [
"message"
],
"term_statistics": true
},
{
"_id": "1"
}
]
}{
"ids": [ "1", "2" ],
"fields": [
"message"
],
"term_statistics": true
}{
"docs": [
{
"_index": "my-index-000001",
"doc" : {
"message" : "test test test"
}
},
{
"_index": "my-index-000001",
"doc" : {
"message" : "Another test ..."
}
}
]
}Copy documents from a source to a destination. You can copy all documents to the destination index or reindex a subset of the documents. The source can be any existing index, alias, or data stream. The destination must differ from the source. For example, you cannot reindex a data stream into itself.
IMPORTANT: Reindex requires _source to be enabled for all documents in the source.
The destination should be configured as wanted before calling the reindex API.
Reindex does not copy the settings from the source or its associated template.
Mappings, shard counts, and replicas, for example, must be configured ahead of time.
If the Elasticsearch security features are enabled, you must have the following security privileges:
read index privilege for the source data stream, index, or alias.write index privilege for the destination data stream, index, or index alias.auto_configure, create_index, or manage index privilege for the destination data stream, index, or alias.source.remote.user must have the monitor cluster privilege and the read index privilege for the source data stream, index, or alias.If reindexing from a remote cluster, you must explicitly allow the remote host in the reindex.remote.whitelist setting.
Automatic data stream creation requires a matching index template with data stream enabled.
The dest element can be configured like the index API to control optimistic concurrency control.
Omitting version_type or setting it to internal causes Elasticsearch to blindly dump documents into the destination, overwriting any that happen to have the same ID.
Setting version_type to external causes Elasticsearch to preserve the version from the source, create any documents that are missing, and update any documents that have an older version in the destination than they do in the source.
Setting op_type to create causes the reindex API to create only missing documents in the destination.
All existing documents will cause a version conflict.
IMPORTANT: Because data streams are append-only, any reindex request to a destination data stream must have an op_type of create.
A reindex can only add new documents to a destination data stream.
It cannot update existing documents in a destination data stream.
By default, version conflicts abort the reindex process.
To continue reindexing if there are conflicts, set the conflicts request body property to proceed.
In this case, the response includes a count of the version conflicts that were encountered.
Note that the handling of other error types is unaffected by the conflicts property.
Additionally, if you opt to count version conflicts, the operation could attempt to reindex more documents from the source than max_docs until it has successfully indexed max_docs documents into the target or it has gone through every document in the source query.
NOTE: The reindex API makes no effort to handle ID collisions. The last document written will "win" but the order isn't usually predictable so it is not a good idea to rely on this behavior. Instead, make sure that IDs are unique by using a script.
Running reindex asynchronously
If the request contains wait_for_completion=false, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task.
Elasticsearch creates a record of this task as a document at _tasks/<task_id>.
Reindex from multiple sources
If you have many sources to reindex it is generally better to reindex them one at a time rather than using a glob pattern to pick up multiple sources. That way you can resume the process if there are any errors by removing the partially completed source and starting over. It also makes parallelizing the process fairly simple: split the list of sources to reindex and run each list in parallel.
For example, you can use a bash script like this:
for index in i1 i2 i3 i4 i5; do
curl -HContent-Type:application/json -XPOST localhost:9200/_reindex?pretty -d'{
"source": {
"index": "'$index'"
},
"dest": {
"index": "'$index'-reindexed"
}
}'
done
Throttling
Set requests_per_second to any positive decimal number (1.4, 6, 1000, for example) to throttle the rate at which reindex issues batches of index operations.
Requests are throttled by padding each batch with a wait time.
To turn off throttling, set requests_per_second to -1.
The throttling is done by waiting between batches so that the scroll that reindex uses internally can be given a timeout that takes into account the padding.
The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing.
By default the batch size is 1000, so if requests_per_second is set to 500:
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
Since the batch is issued as a single bulk request, large batch sizes cause Elasticsearch to create many requests and then wait for a while before starting the next set. This is "bursty" instead of "smooth".
Slicing
Reindex supports sliced scroll to parallelize the reindexing process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.
NOTE: Reindexing from remote clusters does not support manual or automatic slicing.
You can slice a reindex request manually by providing a slice ID and total number of slices to each request.
You can also let reindex automatically parallelize by using sliced scroll to slice on _id.
The slices parameter specifies the number of slices to use.
Adding slices to the reindex request just automates the manual process, creating sub-requests which means it has some quirks:
slices only contains the status of completed slices.slices will rethrottle the unfinished sub-request proportionally.slices will cancel each sub-request.slices, each sub-request won't get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.requests_per_second and max_docs on a request with slices are distributed proportionally to each sub-request. Combine that with the previous point about distribution being uneven and you should conclude that using max_docs with slices might not result in exactly max_docs documents being reindexed.If slicing automatically, setting slices to auto will choose a reasonable number for most indices.
If slicing manually or otherwise tuning automatic slicing, use the following guidelines.
Query performance is most efficient when the number of slices is equal to the number of shards in the index.
If that number is large (for example, 500), choose a lower number as too many slices will hurt performance.
Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.
Indexing performance scales linearly across available resources with the number of slices.
Whether query or indexing performance dominates the runtime depends on the documents being reindexed and cluster resources.
Modify documents during reindexing
Like _update_by_query, reindex operations support a script that modifies the document.
Unlike _update_by_query, the script is allowed to modify the document's metadata.
Just as in _update_by_query, you can set ctx.op to change the operation that is run on the destination.
For example, set ctx.op to noop if your script decides that the document doesn’t have to be indexed in the destination. This "no operation" will be reported in the noop counter in the response body.
Set ctx.op to delete if your script decides that the document must be deleted from the destination.
The deletion will be reported in the deleted counter in the response body.
Setting ctx.op to anything else will return an error, as will setting any other field in ctx.
Think of the possibilities! Just be careful; you are able to change:
_id_index_version_routingSetting _version to null or clearing it from the ctx map is just like not sending the version in an indexing request.
It will cause the document to be overwritten in the destination regardless of the version on the target or the version type you use in the reindex API.
Reindex from remote
Reindex supports reindexing from a remote Elasticsearch cluster.
The host parameter must contain a scheme, host, port, and optional path.
The username and password parameters are optional and when they are present the reindex operation will connect to the remote Elasticsearch node using basic authentication.
Be sure to use HTTPS when using basic authentication or the password will be sent in plain text.
There are a range of settings available to configure the behavior of the HTTPS connection.
When using Elastic Cloud, it is also possible to authenticate against the remote cluster through the use of a valid API key.
Remote hosts must be explicitly allowed with the reindex.remote.whitelist setting.
It can be set to a comma delimited list of allowed remote host and port combinations.
Scheme is ignored; only the host and port are used.
For example:
reindex.remote.whitelist: [otherhost:9200, another:9200, 127.0.10.*:9200, localhost:*"]
The list of allowed hosts must be configured on any nodes that will coordinate the reindex. This feature should work with remote clusters of any version of Elasticsearch. This should enable you to upgrade from any version of Elasticsearch to the current version by reindexing from a cluster of the old version.
WARNING: Elasticsearch does not support forward compatibility across major versions. For example, you cannot reindex from a 7.x cluster into a 6.x cluster.
To enable queries sent to older versions of Elasticsearch, the query parameter is sent directly to the remote host without validation or modification.
NOTE: Reindexing from remote clusters does not support manual or automatic slicing.
Reindexing from a remote server uses an on-heap buffer that defaults to a maximum size of 100mb.
If the remote index includes very large documents you'll need to use a smaller batch size.
It is also possible to set the socket read timeout on the remote connection with the socket_timeout field and the connection timeout with the connect_timeout field.
Both default to 30 seconds.
Configuring SSL parameters
Reindex from remote supports configurable SSL settings.
These must be specified in the elasticsearch.yml file, with the exception of the secure settings, which you add in the Elasticsearch keystore.
It is not possible to configure SSL in the body of the reindex request.
read,writeIf true, the request refreshes affected shards to make this operation visible to search.
The throttle for this request in sub-requests per second. By default, there is no throttle.
The period of time that a consistent view of the index should be maintained for scrolled search.
Values are -1 or 0.
The number of slices this task should be divided into. It defaults to one slice, which means the task isn't sliced into subtasks.
Reindex supports sliced scroll to parallelize the reindexing process. This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.
NOTE: Reindexing from remote clusters does not support manual or automatic slicing.
If set to auto, Elasticsearch chooses the number of slices to use.
This setting will use one slice per shard, up to a certain limit.
If there are multiple sources, it will choose the number of slices based on the index or backing index with the smallest number of shards.
Value is auto.
The maximum number of documents to reindex.
By default, all documents are reindexed.
If it is a value less then or equal to scroll_size, a scroll will not be used to retrieve the results for the operation.
If conflicts is set to proceed, the reindex operation could attempt to reindex more documents from the source than max_docs until it has successfully indexed max_docs documents into the target or it has gone through every document in the source query.
The period each indexing waits for automatic index creation, dynamic mapping updates, and waiting for active shards. By default, Elasticsearch waits for at least one minute before failing. The actual wait time could be longer, particularly when multiple waits occur.
Values are -1 or 0.
The number of shard copies that must be active before proceeding with the operation.
Set it to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
The default value is one, which means it waits for each primary shard to be active.
Values are all or index-setting.
If true, the request blocks until the operation is complete.
If true, the destination must be an index alias.
Values are abort or proceed.
The maximum number of documents to reindex.
By default, all documents are reindexed.
If it is a value less then or equal to scroll_size, a scroll will not be used to retrieve the results for the operation.
If conflicts is set to proceed, the reindex operation could attempt to reindex more documents from the source than max_docs until it has successfully indexed max_docs documents into the target or it has gone through every document in the source query.
POST _reindex
{
"source": {
"index": ["my-index-000001", "my-index-000002"]
},
"dest": {
"index": "my-new-index-000002"
}
}resp = client.reindex(
source={
"index": [
"my-index-000001",
"my-index-000002"
]
},
dest={
"index": "my-new-index-000002"
},
)const response = await client.reindex({
source: {
index: ["my-index-000001", "my-index-000002"],
},
dest: {
index: "my-new-index-000002",
},
});response = client.reindex(
body: {
"source": {
"index": [
"my-index-000001",
"my-index-000002"
]
},
"dest": {
"index": "my-new-index-000002"
}
}
)$resp = $client->reindex([
"body" => [
"source" => [
"index" => array(
"my-index-000001",
"my-index-000002",
),
],
"dest" => [
"index" => "my-new-index-000002",
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"source":{"index":["my-index-000001","my-index-000002"]},"dest":{"index":"my-new-index-000002"}}' "$ELASTICSEARCH_URL/_reindex"client.reindex(r -> r
.dest(d -> d
.index("my-new-index-000002")
)
.source(s -> s
.index(List.of("my-index-000001","my-index-000002"))
)
);
{
"source": {
"index": ["my-index-000001", "my-index-000002"]
},
"dest": {
"index": "my-new-index-000002"
}
}{
"source": {
"index": "metricbeat-*"
},
"dest": {
"index": "metricbeat"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'metricbeat-' + (ctx._index.substring('metricbeat-'.length(), ctx._index.length())) + '-1'"
}
}{
"max_docs": 10,
"source": {
"index": "my-index-000001",
"query": {
"function_score" : {
"random_score" : {},
"min_score" : 0.9
}
}
},
"dest": {
"index": "my-new-index-000001"
}
}{
"source": {
"index": "my-index-000001"
},
"dest": {
"index": "my-new-index-000001",
"version_type": "external"
},
"script": {
"source": "if (ctx._source.foo == 'bar') {ctx._version++; ctx._source.remove('foo')}",
"lang": "painless"
}
}{
"source": {
"remote": {
"host": "http://otherhost:9200",
"username": "user",
"password": "pass"
},
"index": "my-index-000001",
"query": {
"match": {
"test": "data"
}
}
},
"dest": {
"index": "my-new-index-000001"
}
}{
"source": {
"index": "my-index-000001",
"slice": {
"id": 0,
"max": 2
}
},
"dest": {
"index": "my-new-index-000001"
}
}{
"source": {
"index": "my-index-000001"
},
"dest": {
"index": "my-new-index-000001"
}
}{
"source": {
"index": "source",
"query": {
"match": {
"company": "cat"
}
}
},
"dest": {
"index": "dest",
"routing": "=cat"
}
}{
"source": {
"index": "source"
},
"dest": {
"index": "dest",
"pipeline": "some_ingest_pipeline"
}
}{
"source": {
"index": "my-index-000001",
"query": {
"term": {
"user.id": "kimchy"
}
}
},
"dest": {
"index": "my-new-index-000001"
}
}{
"max_docs": 1,
"source": {
"index": "my-index-000001"
},
"dest": {
"index": "my-new-index-000001"
}
}{
"source": {
"index": "my-index-000001",
"_source": ["user.id", "_doc"]
},
"dest": {
"index": "my-new-index-000001"
}
}{
"source": {
"index": "my-index-000001"
},
"dest": {
"index": "my-new-index-000001"
},
"script": {
"source": "ctx._source.tag = ctx._source.remove(\"flag\")"
}
}Change the number of requests per second for a particular reindex operation. For example:
POST _reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
Rethrottling that speeds up the query takes effect immediately. Rethrottling that slows down the query will take effect after completing the current batch. This behavior prevents scroll timeouts.
POST _reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
resp = client.reindex_rethrottle(
task_id="r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second="-1",
)const response = await client.reindexRethrottle({
task_id: "r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second: "-1",
});response = client.reindex_rethrottle(
task_id: "r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second: "-1"
)$resp = $client->reindexRethrottle([
"task_id" => "r1A2WoRbTwKZ516z6NEs5A:36619",
"requests_per_second" => "-1",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1"client.reindexRethrottle(r -> r
.requestsPerSecond(-1.0F)
.taskId("r1A2WoRbTwKZ516z6NEs5A:36619")
);
All methods and paths for this operation:
Get information and statistics about terms in the fields of a particular document.
You can retrieve term vectors for documents stored in the index or for artificial documents passed in the body of the request.
You can specify the fields you are interested in through the fields parameter or by adding the fields to the request body.
For example:
GET /my-index-000001/_termvectors/1?fields=message
Fields can be specified using wildcards, similar to the multi match query.
Term vectors are real-time by default, not near real-time.
This can be changed by setting realtime parameter to false.
You can request three types of values: term information, term statistics, and field statistics. By default, all term information and field statistics are returned for all fields but term statistics are excluded.
Term information
positions: true)offsets: true)payloads: true), as base64 encoded bytesIf the requested information wasn't stored in the index, it will be computed on the fly if possible. Additionally, term vectors could be computed for documents not even existing in the index, but instead provided by the user.
Start and end offsets assume UTF-16 encoding is being used. If you want to use these offsets in order to get the original text that produced this token, you should make sure that the string you are taking a sub-string of is also encoded using UTF-16.
Behaviour
The term and field statistics are not accurate.
Deleted documents are not taken into account.
The information is only retrieved for the shard the requested document resides in.
The term and field statistics are therefore only useful as relative measures whereas the absolute numbers have no meaning in this context.
By default, when requesting term vectors of artificial documents, a shard to get the statistics from is randomly selected.
Use routing only to hit a particular shard.
readThe name of the index that contains the document.
A unique identifier for the document.
A comma-separated list or wildcard expressions of fields to include in the statistics.
It is used as the default list unless a specific field list is provided in the completion_fields or fielddata_fields parameters.
If true, the response includes:
If true, the response includes term offsets.
If true, the response includes term payloads.
If true, the response includes term positions.
The node or shard the operation should be performed on. It is random by default.
If true, the request is real-time as opposed to near-real-time.
A custom value that is used to route operations to a specific shard.
If true, the response includes:
By default these values are not returned since term statistics can have a serious performance impact.
If true, returns the document version as part of a hit.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
An artificial document (a document not present in the index) for which you want to retrieve term vectors.
Override the default per-field analyzer. This is useful in order to generate term vectors in any fashion, especially when using artificial documents. When providing an analyzer for a field that already stores term vectors, the term vectors will be regenerated.
A list of fields to include in the statistics.
It is used as the default list unless a specific field list is provided in the completion_fields or fielddata_fields parameters.
If true, the response includes:
Default value is true.
If true, the response includes term offsets.
Default value is true.
If true, the response includes term payloads.
Default value is true.
If true, the response includes term positions.
Default value is true.
If true, the response includes:
By default these values are not returned since term statistics can have a serious performance impact.
Default value is false.
Values are internal, external, external_gte, or force.
GET /my-index-000001/_termvectors/1
{
"fields" : ["text"],
"offsets" : true,
"payloads" : true,
"positions" : true,
"term_statistics" : true,
"field_statistics" : true
}resp = client.termvectors(
index="my-index-000001",
id="1",
fields=[
"text"
],
offsets=True,
payloads=True,
positions=True,
term_statistics=True,
field_statistics=True,
)const response = await client.termvectors({
index: "my-index-000001",
id: 1,
fields: ["text"],
offsets: true,
payloads: true,
positions: true,
term_statistics: true,
field_statistics: true,
});response = client.termvectors(
index: "my-index-000001",
id: "1",
body: {
"fields": [
"text"
],
"offsets": true,
"payloads": true,
"positions": true,
"term_statistics": true,
"field_statistics": true
}
)$resp = $client->termvectors([
"index" => "my-index-000001",
"id" => "1",
"body" => [
"fields" => array(
"text",
),
"offsets" => true,
"payloads" => true,
"positions" => true,
"term_statistics" => true,
"field_statistics" => true,
],
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"fields":["text"],"offsets":true,"payloads":true,"positions":true,"term_statistics":true,"field_statistics":true}' "$ELASTICSEARCH_URL/my-index-000001/_termvectors/1"client.termvectors(t -> t
.fieldStatistics(true)
.fields("text")
.id("1")
.index("my-index-000001")
.offsets(true)
.payloads(true)
.positions(true)
.termStatistics(true)
);
{
"fields" : ["text"],
"offsets" : true,
"payloads" : true,
"positions" : true,
"term_statistics" : true,
"field_statistics" : true
}{
"doc" : {
"fullname" : "John Doe",
"text" : "test test test"
},
"fields": ["fullname"],
"per_field_analyzer" : {
"fullname": "keyword"
}
}{
"doc": {
"plot": "When wealthy industrialist Tony Stark is forced to build an armored suit after a life-threatening incident, he ultimately decides to use its technology to fight against evil."
},
"term_statistics": true,
"field_statistics": true,
"positions": false,
"offsets": false,
"filter": {
"max_num_terms": 3,
"min_term_freq": 1,
"min_doc_freq": 1
}
}{
"fields" : ["text", "some_field_without_term_vectors"],
"offsets" : true,
"positions" : true,
"term_statistics" : true,
"field_statistics" : true
}{
"doc" : {
"fullname" : "John Doe",
"text" : "test test test"
}
}{
"_index": "my-index-000001",
"_id": "1",
"_version": 1,
"found": true,
"took": 6,
"term_vectors": {
"text": {
"field_statistics": {
"sum_doc_freq": 4,
"doc_count": 2,
"sum_ttf": 6
},
"terms": {
"test": {
"doc_freq": 2,
"ttf": 4,
"term_freq": 3,
"tokens": [
{
"position": 0,
"start_offset": 0,
"end_offset": 4,
"payload": "d29yZA=="
},
{
"position": 1,
"start_offset": 5,
"end_offset": 9,
"payload": "d29yZA=="
},
{
"position": 2,
"start_offset": 10,
"end_offset": 14,
"payload": "d29yZA=="
}
]
}
}
}
}
}{
"_index": "my-index-000001",
"_version": 0,
"found": true,
"took": 6,
"term_vectors": {
"fullname": {
"field_statistics": {
"sum_doc_freq": 2,
"doc_count": 4,
"sum_ttf": 4
},
"terms": {
"John Doe": {
"term_freq": 1,
"tokens": [
{
"position": 0,
"start_offset": 0,
"end_offset": 8
}
]
}
}
}
}
}{
"_index": "imdb",
"_version": 0,
"found": true,
"term_vectors": {
"plot": {
"field_statistics": {
"sum_doc_freq": 3384269,
"doc_count": 176214,
"sum_ttf": 3753460
},
"terms": {
"armored": {
"doc_freq": 27,
"ttf": 27,
"term_freq": 1,
"score": 9.74725
},
"industrialist": {
"doc_freq": 88,
"ttf": 88,
"term_freq": 1,
"score": 8.590818
},
"stark": {
"doc_freq": 44,
"ttf": 47,
"term_freq": 1,
"score": 9.272792
}
}
}
}
}Updates documents that match the specified query. If no query is specified, performs an update on every document in the data stream or index without modifying the source, which is useful for picking up mapping changes.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or alias:
readindex or writeYou can specify the query criteria in the request URI or the request body using the same syntax as the search API.
When you submit an update by query request, Elasticsearch gets a snapshot of the data stream or index when it begins processing the request and updates matching documents using internal versioning.
When the versions match, the document is updated and the version number is incremented.
If a document changes between the time that the snapshot is taken and the update operation is processed, it results in a version conflict and the operation fails.
You can opt to count version conflicts instead of halting and returning by setting conflicts to proceed.
Note that if you opt to count version conflicts, the operation could attempt to update more documents from the source than max_docs until it has successfully updated max_docs documents or it has gone through every document in the source query.
NOTE: Documents with a version equal to 0 cannot be updated using update by query because internal versioning does not support 0 as a valid version number.
While processing an update by query request, Elasticsearch performs multiple search requests sequentially to find all of the matching documents. A bulk update request is performed for each batch of matching documents. Any query or update failures cause the update by query request to fail and the failures are shown in the response. Any update requests that completed successfully still stick, they are not rolled back.
Throttling update requests
To control the rate at which update by query issues batches of update operations, you can set requests_per_second to any positive decimal number.
This pads each batch with a wait time to throttle the rate.
Set requests_per_second to -1 to turn off throttling.
Throttling uses a wait time between batches so that the internal scroll requests can be given a timeout that takes the request padding into account.
The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing.
By default the batch size is 1000, so if requests_per_second is set to 500:
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
Since the batch is issued as a single _bulk request, large batch sizes cause Elasticsearch to create many requests and wait before starting the next set. This is "bursty" instead of "smooth".
Slicing
Update by query supports sliced scroll to parallelize the update process. This can improve efficiency and provide a convenient way to break the request down into smaller parts.
Setting slices to auto chooses a reasonable number for most data streams and indices.
This setting will use one slice per shard, up to a certain limit.
If there are multiple source data streams or indices, it will choose the number of slices based on the index or backing index with the smallest number of shards.
Adding slices to _update_by_query just automates the manual process of creating sub-requests, which means it has some quirks:
slices only contains the status of completed slices.slices will rethrottle the unfinished sub-request proportionally.requests_per_second and max_docs on a request with slices are distributed proportionally to each sub-request. Combine that with the point above about distribution being uneven and you should conclude that using max_docs with slices might not result in exactly max_docs documents being updated.If you're slicing manually or otherwise tuning automatic slicing, keep in mind that:
Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources.
Update the document source
Update by query supports scripts to update the document source.
As with the update API, you can set ctx.op to change the operation that is performed.
Set ctx.op = "noop" if your script decides that it doesn't have to make any changes.
The update by query operation skips updating the document and increments the noop counter.
Set ctx.op = "delete" if your script decides that the document should be deleted.
The update by query operation deletes the document and increments the deleted counter.
Update by query supports only index, noop, and delete.
Setting ctx.op to anything else is an error.
Setting any other field in ctx is an error.
This API enables you to only modify the source of matching documents; you cannot move them.
read,writeA comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (*).
To search all data streams or indices, omit this parameter or use * or _all.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.
The analyzer to use for the query string.
This parameter can be used only when the q query string parameter is specified.
If true, wildcard and prefix queries are analyzed.
This parameter can be used only when the q query string parameter is specified.
The preferred behavior when update by query hits version conflicts: abort or proceed.
Supported values include:
abort: Stop reindexing if there are conflicts.proceed: Continue reindexing even if there are conflicts.Values are abort or proceed.
The default operator for query string query: AND or OR.
This parameter can be used only when the q query string parameter is specified.
Values are and, AND, or, or OR.
The field to use as default where no field prefix is given in the query string.
This parameter can be used only when the q query string parameter is specified.
The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
It supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Skips the specified number of documents.
If true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
This parameter can be used only when the q query string parameter is specified.
The maximum number of documents to process.
It defaults to all documents.
When set to a value less then or equal to scroll_size then a scroll will not be used to retrieve the results for the operation.
The ID of the pipeline to use to preprocess incoming documents.
If the index has a default ingest pipeline specified, then setting the value to _none disables the default ingest pipeline for this request.
If a final pipeline is configured it will always run, regardless of the value of this parameter.
The node or shard the operation should be performed on. It is random by default.
A query in the Lucene query string syntax.
If true, Elasticsearch refreshes affected shards to make the operation visible to search after the request completes.
This is different than the update API's refresh parameter, which causes just the shard that received the request to be refreshed.
If true, the request cache is used for this request.
It defaults to the index-level setting.
The throttle for this request in sub-requests per second.
A custom value used to route operations to a specific shard.
The period to retain the search context for scrolling.
Values are -1 or 0.
The size of the scroll request that powers the operation.
An explicit timeout for each search request. By default, there is no timeout.
Values are -1 or 0.
The type of the search operation. Available options include query_then_fetch and dfs_query_then_fetch.
Supported values include:
query_then_fetch: Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.dfs_query_then_fetch: Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.Values are query_then_fetch or dfs_query_then_fetch.
The number of slices this task should be divided into.
Value is auto.
A comma-separated list of : pairs.
The specific tag of the request for logging and statistical purposes.
The maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.
IMPORTANT: Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.
The period each update request waits for the following operations: dynamic mapping updates, waiting for active shards. By default, it is one minute. This guarantees Elasticsearch waits for at least the timeout before failing. The actual wait time could be longer, particularly when multiple waits occur.
Values are -1 or 0.
If true, returns the document version as part of a hit.
Should the document increment the version number (internal) on hit or not (reindex)
The number of shard copies that must be active before proceeding with the operation.
Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
The timeout parameter controls how long each write request waits for unavailable shards to become available.
Both work exactly the way they work in the bulk API.
Values are all or index-setting.
If true, the request blocks until the operation is complete.
If false, Elasticsearch performs some preflight checks, launches the request, and returns a task ID that you can use to cancel or get the status of the task.
Elasticsearch creates a record of this task as a document at .tasks/task/${taskId}.
The maximum number of documents to update.
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
Values are abort or proceed.
POST my-index-000001/_update_by_query?conflicts=proceed
{
"query": {
"term": {
"user.id": "kimchy"
}
}
}resp = client.update_by_query(
index="my-index-000001",
conflicts="proceed",
query={
"term": {
"user.id": "kimchy"
}
},
)const response = await client.updateByQuery({
index: "my-index-000001",
conflicts: "proceed",
query: {
term: {
"user.id": "kimchy",
},
},
});response = client.update_by_query(
index: "my-index-000001",
conflicts: "proceed",
body: {
"query": {
"term": {
"user.id": "kimchy"
}
}
}
)$resp = $client->updateByQuery([
"index" => "my-index-000001",
"conflicts" => "proceed",
"body" => [
"query" => [
"term" => [
"user.id" => "kimchy",
],
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"query":{"term":{"user.id":"kimchy"}}}' "$ELASTICSEARCH_URL/my-index-000001/_update_by_query?conflicts=proceed"client.updateByQuery(u -> u
.conflicts(Conflicts.Proceed)
.index("my-index-000001")
.query(q -> q
.term(t -> t
.field("user.id")
.value(FieldValue.of("kimchy"))
)
)
);
{
"query": {
"term": {
"user.id": "kimchy"
}
}
}{
"script": {
"source": "ctx._source.count++",
"lang": "painless"
},
"query": {
"term": {
"user.id": "kimchy"
}
}
}{
"slice": {
"id": 0,
"max": 2
},
"script": {
"source": "ctx._source['extra'] = 'test'"
}
}{
"script": {
"source": "ctx._source['extra'] = 'test'"
}
}Change the number of requests per second for a particular update by query operation. Rethrottling that speeds up the query takes effect immediately but rethrotting that slows down the query takes effect after completing the current batch to prevent scroll timeouts.
POST _update_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
resp = client.update_by_query_rethrottle(
task_id="r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second="-1",
)const response = await client.updateByQueryRethrottle({
task_id: "r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second: "-1",
});response = client.update_by_query_rethrottle(
task_id: "r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second: "-1"
)$resp = $client->updateByQueryRethrottle([
"task_id" => "r1A2WoRbTwKZ516z6NEs5A:36619",
"requests_per_second" => "-1",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_update_by_query/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1"client.updateByQueryRethrottle(u -> u
.requestsPerSecond(-1.0F)
.taskId("r1A2WoRbTwKZ516z6NEs5A:36619")
);
All methods and paths for this operation:
Returns information about an enrich policy.
Comma-separated list of enrich policy names used to limit the request. To return information for all enrich policies, omit this parameter.
GET /_enrich/policy/my-policy
resp = client.enrich.get_policy(
name="my-policy",
)const response = await client.enrich.getPolicy({
name: "my-policy",
});response = client.enrich.get_policy(
name: "my-policy"
)$resp = $client->enrich()->getPolicy([
"name" => "my-policy",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_enrich/policy/my-policy"client.enrich().getPolicy(g -> g
.name("my-policy")
);
Creates an enrich policy.
PUT /_enrich/policy/postal_policy
{
"geo_match": {
"indices": "postal_codes",
"match_field": "location",
"enrich_fields": [ "location", "postal_code" ]
}
}resp = client.enrich.put_policy(
name="postal_policy",
geo_match={
"indices": "postal_codes",
"match_field": "location",
"enrich_fields": [
"location",
"postal_code"
]
},
)const response = await client.enrich.putPolicy({
name: "postal_policy",
geo_match: {
indices: "postal_codes",
match_field: "location",
enrich_fields: ["location", "postal_code"],
},
});response = client.enrich.put_policy(
name: "postal_policy",
body: {
"geo_match": {
"indices": "postal_codes",
"match_field": "location",
"enrich_fields": [
"location",
"postal_code"
]
}
}
)$resp = $client->enrich()->putPolicy([
"name" => "postal_policy",
"body" => [
"geo_match" => [
"indices" => "postal_codes",
"match_field" => "location",
"enrich_fields" => array(
"location",
"postal_code",
),
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"geo_match":{"indices":"postal_codes","match_field":"location","enrich_fields":["location","postal_code"]}}' "$ELASTICSEARCH_URL/_enrich/policy/postal_policy"client.enrich().putPolicy(p -> p
.geoMatch(g -> g
.enrichFields(List.of("location","postal_code"))
.indices("postal_codes")
.matchField("location")
)
.name("postal_policy")
);
{
"geo_match": {
"indices": "postal_codes",
"match_field": "location",
"enrich_fields": [ "location", "postal_code" ]
}
}Deletes an existing enrich policy and its enrich index.
DELETE /_enrich/policy/my-policy
resp = client.enrich.delete_policy(
name="my-policy",
)const response = await client.enrich.deletePolicy({
name: "my-policy",
});response = client.enrich.delete_policy(
name: "my-policy"
)$resp = $client->enrich()->deletePolicy([
"name" => "my-policy",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_enrich/policy/my-policy"client.enrich().deletePolicy(d -> d
.name("my-policy")
);
Returns enrich coordinator statistics and information about enrich policies that are currently executing.
GET /_enrich/_stats
resp = client.enrich.stats()const response = await client.enrich.stats();response = client.enrich.stats$resp = $client->enrich()->stats();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_enrich/_stats"client.enrich().stats(s -> s);
The Elasticsearch Query Language (ES|QL) provides a powerful way to filter, transform, and analyze data stored in Elasticsearch, and in the future in other runtimes.
Get a list of features that can be included in snapshots using the feature_states field when creating a snapshot.
You can use this API to determine which feature states to include when taking a snapshot.
By default, all feature states are included in a snapshot if that snapshot includes the global state, or none if it does not.
A feature state includes one or more system indices necessary for a given feature to function. In order to ensure data integrity, all system indices that comprise a feature state are snapshotted and restored together.
The features listed by this API are a combination of built-in features and features defined by plugins. In order for a feature state to be listed in this API and recognized as a valid feature state by the create snapshot API, the plugin that defines that feature must be installed on the master node.
GET _features
resp = client.features.get_features()const response = await client.features.getFeatures();response = client.features.get_features$resp = $client->features()->getFeatures();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_features"client.features().getFeatures(g -> g);
{
"features": [
{
"name": "tasks",
"description": "Manages task results"
},
{
"name": "kibana",
"description": "Manages Kibana configuration and reports"
}
]
}Get the current global checkpoints for an index. This API is designed for internal use by the Fleet server project.
A boolean value which controls whether to wait (until the timeout) for the global checkpoints
to advance past the provided checkpoints.
A boolean value which controls whether to wait (until the timeout) for the target index to exist
and all primary shards be active. Can only be true when wait_for_advance is true.
A comma separated list of previous global checkpoints. When used in combination with wait_for_advance,
the API will only return once the global checkpoints advances past the checkpoints. Providing an empty list
will cause Elasticsearch to immediately return the current global checkpoints.
Period to wait for a global checkpoints to advance past checkpoints.
Values are -1 or 0.
curl \
--request GET 'http://api.example.com/{index}/_fleet/global_checkpoints' \
--header "Authorization: $API_KEY"The graph explore API enables you to extract and summarize information about the documents and terms in an Elasticsearch data stream or index.
All methods and paths for this operation:
Extract and summarize information about the documents and terms in an Elasticsearch data stream or index.
The easiest way to understand the behavior of this API is to use the Graph UI to explore connections.
An initial request to the _explore API contains a seed query that identifies the documents of interest and specifies the fields that define the vertices and connections you want to include in the graph.
Subsequent requests enable you to spider out from one more vertices of interest.
You can exclude vertices that have already been returned.
Custom value used to route operations to a specific shard.
Specifies the period of time to wait for a response from each shard. If no response is received before the timeout expires, the request fails and returns an error. Defaults to no timeout.
Values are -1 or 0.
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
Specifies one or more fields that contain the terms you want to include in the graph as vertices.
POST clicklogs/_graph/explore
{
"query": {
"match": {
"query.raw": "midi"
}
},
"vertices": [
{
"field": "product"
}
],
"connections": {
"vertices": [
{
"field": "query.raw"
}
]
}
}resp = client.graph.explore(
index="clicklogs",
query={
"match": {
"query.raw": "midi"
}
},
vertices=[
{
"field": "product"
}
],
connections={
"vertices": [
{
"field": "query.raw"
}
]
},
)const response = await client.graph.explore({
index: "clicklogs",
query: {
match: {
"query.raw": "midi",
},
},
vertices: [
{
field: "product",
},
],
connections: {
vertices: [
{
field: "query.raw",
},
],
},
});response = client.graph.explore(
index: "clicklogs",
body: {
"query": {
"match": {
"query.raw": "midi"
}
},
"vertices": [
{
"field": "product"
}
],
"connections": {
"vertices": [
{
"field": "query.raw"
}
]
}
}
)$resp = $client->graph()->explore([
"index" => "clicklogs",
"body" => [
"query" => [
"match" => [
"query.raw" => "midi",
],
],
"vertices" => array(
[
"field" => "product",
],
),
"connections" => [
"vertices" => array(
[
"field" => "query.raw",
],
),
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"query":{"match":{"query.raw":"midi"}},"vertices":[{"field":"product"}],"connections":{"vertices":[{"field":"query.raw"}]}}' "$ELASTICSEARCH_URL/clicklogs/_graph/explore"client.graph().explore(e -> e
.connections(c -> c
.vertices(v -> v
.field("query.raw")
)
)
.index("clicklogs")
.query(q -> q
.match(m -> m
.field("query.raw")
.query(FieldValue.of("midi"))
)
)
.vertices(v -> v
.field("product")
)
);
{
"query": {
"match": {
"query.raw": "midi"
}
},
"vertices": [
{
"field": "product"
}
],
"connections": {
"vertices": [
{
"field": "query.raw"
}
]
}
}Comma-separated list or wildcard expression of component template names used to limit the request.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
DELETE _component_template/template_1
resp = client.cluster.delete_component_template(
name="template_1",
)const response = await client.cluster.deleteComponentTemplate({
name: "template_1",
});response = client.cluster.delete_component_template(
name: "template_1"
)$resp = $client->cluster()->deleteComponentTemplate([
"name" => "template_1",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_component_template/template_1"client.cluster().deleteComponentTemplate(d -> d
.name("template_1")
);
If Elasticsearch encounters index data that is absent from the current cluster state, those indices are considered to be dangling.
For example, this can happen if you delete more than cluster.indices.tombstones.size indices while an Elasticsearch node is offline.
manageDELETE /_dangling/<index-uuid>?accept_data_loss=true
resp = client.dangling_indices.delete_dangling_index(
index_uuid="<index-uuid>",
accept_data_loss=True,
)const response = await client.danglingIndices.deleteDanglingIndex({
index_uuid: "<index-uuid>",
accept_data_loss: "true",
});response = client.dangling_indices.delete_dangling_index(
index_uuid: "<index-uuid>",
accept_data_loss: "true"
)$resp = $client->danglingIndices()->deleteDanglingIndex([
"index_uuid" => "<index-uuid>",
"accept_data_loss" => "true",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_dangling/<index-uuid>?accept_data_loss=true"client.danglingIndices().deleteDanglingIndex(d -> d
.acceptDataLoss(true)
.indexUuid("<index-uuid>")
);
All methods and paths for this operation:
Clone an existing index into a new index. Each original primary shard is cloned into a new primary shard in the new index.
IMPORTANT: Elasticsearch does not apply index templates to the resulting index. The API also does not copy index metadata from the original index. Index metadata includes aliases, index lifecycle management phase definitions, and cross-cluster replication (CCR) follower information. For example, if you clone a CCR follower index, the resulting clone will not be a follower index.
The clone API copies most index settings from the source index to the resulting index, with the exception of index.number_of_replicas and index.auto_expand_replicas.
To set the number of replicas in the resulting index, configure these settings in the clone request.
Cloning works as follows:
IMPORTANT: Indices can only be cloned if they meet the following requirements:
The current write index on a data stream cannot be cloned. In order to clone the current write index, the data stream must first be rolled over so that a new write index is created and then the previous write index can be cloned.
NOTE: Mappings cannot be specified in the _clone request. The mappings of the source index will be used for the target index.
Monitor the cloning process
The cloning process can be monitored with the cat recovery API or the cluster health API can be used to wait until all primary shards have been allocated by setting the wait_for_status parameter to yellow.
The _clone API returns as soon as the target index has been added to the cluster state, before any shards have been allocated.
At this point, all shards are in the state unassigned.
If, for any reason, the target index can't be allocated, its primary shard will remain unassigned until it can be allocated on that node.
Once the primary shard is allocated, it moves to state initializing, and the clone process begins. When the clone operation completes, the shard will become active. At that point, Elasticsearch will try to allocate any replicas and may decide to relocate the primary shard to another node.
Wait for active shards
Because the clone operation creates a new index to clone the shards to, the wait for active shards setting on index creation applies to the clone index action as well.
managePeriod to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
The number of shard copies that must be active before proceeding with the operation.
Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
Values are all or index-setting.
POST /my_source_index/_clone/my_target_index
{
"settings": {
"index.number_of_shards": 5
},
"aliases": {
"my_search_indices": {}
}
}resp = client.indices.clone(
index="my_source_index",
target="my_target_index",
settings={
"index.number_of_shards": 5
},
aliases={
"my_search_indices": {}
},
)const response = await client.indices.clone({
index: "my_source_index",
target: "my_target_index",
settings: {
"index.number_of_shards": 5,
},
aliases: {
my_search_indices: {},
},
});response = client.indices.clone(
index: "my_source_index",
target: "my_target_index",
body: {
"settings": {
"index.number_of_shards": 5
},
"aliases": {
"my_search_indices": {}
}
}
)$resp = $client->indices()->clone([
"index" => "my_source_index",
"target" => "my_target_index",
"body" => [
"settings" => [
"index.number_of_shards" => 5,
],
"aliases" => [
"my_search_indices" => new ArrayObject([]),
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"settings":{"index.number_of_shards":5},"aliases":{"my_search_indices":{}}}' "$ELASTICSEARCH_URL/my_source_index/_clone/my_target_index"client.indices().clone(c -> c
.aliases("my_search_indices", a -> a)
.index("my_source_index")
.settings("index.number_of_shards", JsonData.fromJson("5"))
.target("my_target_index")
);
{
"settings": {
"index.number_of_shards": 5
},
"aliases": {
"my_search_indices": {}
}
}You can use the create index API to add a new index to an Elasticsearch cluster. When creating an index, you can specify the following:
Wait for active shards
By default, index creation will only return a response to the client when the primary copies of each shard have been started, or the request times out.
The index creation response will indicate what happened.
For example, acknowledged indicates whether the index was successfully created in the cluster, while shards_acknowledged indicates whether the requisite number of shard copies were started for each shard in the index before timing out.
Note that it is still possible for either acknowledged or shards_acknowledged to be false, but for the index creation to be successful.
These values simply indicate whether the operation completed before the timeout.
If acknowledged is false, the request timed out before the cluster state was updated with the newly created index, but it probably will be created sometime soon.
If shards_acknowledged is false, then the request timed out before the requisite number of shards were started (by default just the primaries), even if the cluster state was successfully updated to reflect the newly created index (that is to say, acknowledged is true).
You can change the default of only waiting for the primary shards to start through the index setting index.write.wait_for_active_shards.
Note that changing this setting will also affect the wait_for_active_shards value on all subsequent write operations.
create_index,managePeriod to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
The number of shard copies that must be active before proceeding with the operation.
Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
Values are all or index-setting.
Aliases for the index.
PUT /my-index-000001
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}resp = client.indices.create(
index="my-index-000001",
settings={
"number_of_shards": 3,
"number_of_replicas": 2
},
)const response = await client.indices.create({
index: "my-index-000001",
settings: {
number_of_shards: 3,
number_of_replicas: 2,
},
});response = client.indices.create(
index: "my-index-000001",
body: {
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}
)$resp = $client->indices()->create([
"index" => "my-index-000001",
"body" => [
"settings" => [
"number_of_shards" => 3,
"number_of_replicas" => 2,
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"settings":{"number_of_shards":3,"number_of_replicas":2}}' "$ELASTICSEARCH_URL/my-index-000001"client.indices().create(c -> c
.index("my-index-000001")
.settings(s -> s
.numberOfShards("3")
.numberOfReplicas("2")
)
);
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}{
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"field1": { "type": "text" }
}
}
}{
"aliases": {
"alias_1": {},
"alias_2": {
"filter": {
"term": {
"user.id": "kimchy"
}
},
"routing": "shard-1"
}
}
}Check if one or more indices, index aliases, or data streams exist.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, returns settings in flat format.
If true, return all default settings in the response.
If true, the request retrieves information from the local node only.
HEAD my-data-stream
resp = client.indices.exists(
index="my-data-stream",
)const response = await client.indices.exists({
index: "my-data-stream",
});response = client.indices.exists(
index: "my-data-stream"
)$resp = $client->indices()->exists([
"index" => "my-data-stream",
]);curl --head -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-data-stream"client.indices().exists(e -> e
.index("my-data-stream")
);
All methods and paths for this operation:
Adds a data stream or index to an alias.
Comma-separated list of data streams or indices to add.
Supports wildcards (*).
Wildcard patterns that match both data streams and indices return an error.
Alias to update. If the alias doesn’t exist, the request creates it. Index alias names support date math.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
If true, sets the write index or data stream for the alias.
If an alias points to multiple indices or data streams and is_write_index isn’t set, the alias rejects write requests.
If an index alias points to one index and is_write_index isn’t set, the index automatically acts as the write index.
Data stream aliases don’t automatically set a write data stream, even if the alias points to one data stream.
POST _aliases
{
"actions": [
{
"add": {
"index": "my-data-stream",
"alias": "my-alias"
}
}
]
}resp = client.indices.update_aliases(
actions=[
{
"add": {
"index": "my-data-stream",
"alias": "my-alias"
}
}
],
)const response = await client.indices.updateAliases({
actions: [
{
add: {
index: "my-data-stream",
alias: "my-alias",
},
},
],
});response = client.indices.update_aliases(
body: {
"actions": [
{
"add": {
"index": "my-data-stream",
"alias": "my-alias"
}
}
]
}
)$resp = $client->indices()->updateAliases([
"body" => [
"actions" => array(
[
"add" => [
"index" => "my-data-stream",
"alias" => "my-alias",
],
],
),
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"actions":[{"add":{"index":"my-data-stream","alias":"my-alias"}}]}' "$ELASTICSEARCH_URL/_aliases"client.indices().updateAliases(u -> u
.actions(a -> a
.add(ad -> ad
.alias("my-alias")
.index("my-data-stream")
)
)
);
{
"actions": [
{
"add": {
"index": "my-data-stream",
"alias": "my-alias"
}
}
]
}Comma-separated list of data streams or indices used to limit the request.
Supports wildcards (*).
Comma-separated list of aliases to remove.
Supports wildcards (*). To remove all aliases, use * or _all.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
DELETE my-data-stream/_alias/my-alias
resp = client.indices.delete_alias(
index="my-data-stream",
name="my-alias",
)const response = await client.indices.deleteAlias({
index: "my-data-stream",
name: "my-alias",
});response = client.indices.delete_alias(
index: "my-data-stream",
name: "my-alias"
)$resp = $client->indices()->deleteAlias([
"index" => "my-data-stream",
"name" => "my-alias",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-data-stream/_alias/my-alias"client.indices().deleteAlias(d -> d
.index("my-data-stream")
.name("my-alias")
);
Removes the data stream lifecycle from a data stream, rendering it not managed by the data stream lifecycle.
A comma-separated list of data streams of which the data stream lifecycle will be deleted; use * to get all data streams
Whether wildcard expressions should get expanded to open or closed indices (default: open)
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Specify timeout for connection to master
Values are -1 or 0.
Explicit timestamp for the document
Values are -1 or 0.
DELETE _data_stream/my-data-stream/_lifecycle
resp = client.indices.delete_data_lifecycle(
name="my-data-stream",
)const response = await client.indices.deleteDataLifecycle({
name: "my-data-stream",
});response = client.indices.delete_data_lifecycle(
name: "my-data-stream"
)$resp = $client->indices()->deleteDataLifecycle([
"name" => "my-data-stream",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/my-data-stream/_lifecycle"client.indices().deleteDataLifecycle(d -> d
.name("my-data-stream")
);
{
"acknowledged": true
}All methods and paths for this operation:
Index templates define settings, mappings, and aliases that can be applied automatically to new indices.
Elasticsearch applies templates to new indices based on an wildcard pattern that matches the index name. Index templates are applied during data stream or index creation. For data streams, these settings and mappings are applied when the stream's backing indices are created. Settings and mappings specified in a create index API request override any settings or mappings specified in an index template. Changes to index templates do not affect existing indices, including the existing backing indices of a data stream.
You can use C-style /* *\/ block comments in index templates.
You can include comments anywhere in the request body, except before the opening curly bracket.
Multiple matching templates
If multiple index templates match the name of a new index or data stream, the template with the highest priority is used.
Multiple templates with overlapping index patterns at the same priority are not allowed and an error will be thrown when attempting to create a template matching an existing index template at identical priorities.
Composing aliases, mappings, and settings
When multiple component templates are specified in the composed_of field for an index template, they are merged in the order specified, meaning that later component templates override earlier component templates.
Any mappings, settings, or aliases from the parent index template are merged in next.
Finally, any configuration on the index request itself is merged.
Mapping definitions are merged recursively, which means that later mapping components can introduce new field mappings and update the mapping configuration.
If a field mapping is already contained in an earlier component, its definition will be completely overwritten by the later one.
This recursive merging strategy applies not only to field mappings, but also root options like dynamic_templates and meta.
If an earlier component contains a dynamic_templates block, then by default new dynamic_templates entries are appended onto the end.
If an entry already exists with the same key, then it is overwritten by the new definition.
manage_index_templatesIf true, this request cannot replace or update existing index templates.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
User defined reason for creating/updating the index template
An ordered list of component template names. Component templates are merged in the order specified, meaning that the last component template specified has the highest precedence.
Priority to determine index template precedence when a new data stream or index is created. The index template with the highest priority is chosen. If no priority is specified the template is treated as though it is of priority 0 (lowest priority). This number is not automatically generated by Elasticsearch.
This setting overrides the value of the action.auto_create_index cluster setting.
If set to true in a template, then indices can be automatically created using that template even if auto-creation of indices is disabled via actions.auto_create_index.
If set to false, then indices or data streams matching the template must always be explicitly created, and may never be automatically created.
The configuration option ignore_missing_component_templates can be used when an index template references a component template that might not exist
Marks this index template as deprecated. When creating or updating a non-deprecated index template that uses deprecated components, Elasticsearch will emit a deprecation warning.
PUT /_index_template/template_1
{
"index_patterns" : ["template*"],
"priority" : 1,
"template": {
"settings" : {
"number_of_shards" : 2
}
}
}resp = client.indices.put_index_template(
name="template_1",
index_patterns=[
"template*"
],
priority=1,
template={
"settings": {
"number_of_shards": 2
}
},
)const response = await client.indices.putIndexTemplate({
name: "template_1",
index_patterns: ["template*"],
priority: 1,
template: {
settings: {
number_of_shards: 2,
},
},
});response = client.indices.put_index_template(
name: "template_1",
body: {
"index_patterns": [
"template*"
],
"priority": 1,
"template": {
"settings": {
"number_of_shards": 2
}
}
}
)$resp = $client->indices()->putIndexTemplate([
"name" => "template_1",
"body" => [
"index_patterns" => array(
"template*",
),
"priority" => 1,
"template" => [
"settings" => [
"number_of_shards" => 2,
],
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"index_patterns":["template*"],"priority":1,"template":{"settings":{"number_of_shards":2}}}' "$ELASTICSEARCH_URL/_index_template/template_1"client.indices().putIndexTemplate(p -> p
.indexPatterns("template*")
.name("template_1")
.priority(1L)
.template(t -> t
.settings(s -> s
.numberOfShards("2")
)
)
);
{
"index_patterns" : ["template*"],
"priority" : 1,
"template": {
"settings" : {
"number_of_shards" : 2
}
}
}{
"index_patterns": [
"template*"
],
"template": {
"settings": {
"number_of_shards": 1
},
"aliases": {
"alias1": {},
"alias2": {
"filter": {
"term": {
"user.id": "kimchy"
}
},
"routing": "shard-1"
},
"{index}-alias": {}
}
}
}Comma-separated list of index template names used to limit the request. Wildcard (*) expressions are supported.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
DELETE /_index_template/my-index-template
resp = client.indices.delete_index_template(
name="my-index-template",
)const response = await client.indices.deleteIndexTemplate({
name: "my-index-template",
});response = client.indices.delete_index_template(
name: "my-index-template"
)$resp = $client->indices()->deleteIndexTemplate([
"name" => "my-index-template",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_index_template/my-index-template"client.indices().deleteIndexTemplate(d -> d
.name("my-index-template")
);
The name of the legacy index template to delete.
Wildcard (*) expressions are supported.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
DELETE _template/.cloud-hot-warm-allocation-0
resp = client.indices.delete_template(
name=".cloud-hot-warm-allocation-0",
)const response = await client.indices.deleteTemplate({
name: ".cloud-hot-warm-allocation-0",
});response = client.indices.delete_template(
name: ".cloud-hot-warm-allocation-0"
)$resp = $client->indices()->deleteTemplate([
"name" => ".cloud-hot-warm-allocation-0",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_template/.cloud-hot-warm-allocation-0"client.indices().deleteTemplate(d -> d
.name(".cloud-hot-warm-allocation-0")
);
Get field usage information for each shard and field of an index. Field usage statistics are automatically captured when queries are running on a cluster. A shard-level search request that accesses a given field, even if multiple times during that request, is counted as a single use.
The response body reports the per-shard usage count of the data structures that back the fields in the index. A given request will increment each count by a maximum value of 1, even if the request accesses the same field multiple times.
manageComma-separated list or wildcard expression of index names used to limit the request.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Comma-separated list or wildcard expressions of fields to include in the statistics.
GET /my-index-000001/_field_usage_stats
resp = client.indices.field_usage_stats(
index="my-index-000001",
)const response = await client.indices.fieldUsageStats({
index: "my-index-000001",
});response = client.indices.field_usage_stats(
index: "my-index-000001"
)$resp = $client->indices()->fieldUsageStats([
"index" => "my-index-000001",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_field_usage_stats"client.indices().fieldUsageStats(f -> f
.index("my-index-000001")
);
{
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"my-index-000001": {
"shards": [
{
"tracking_id": "MpOl0QlTQ4SYYhEe6KgJoQ",
"tracking_started_at_millis": 1625558985010,
"routing": {
"state": "STARTED",
"primary": true,
"node": "gA6KeeVzQkGURFCUyV-e8Q",
"relocating_node": null
},
"stats": {
"all_fields": {
"any": "6",
"inverted_index": {
"terms": 1,
"postings": 1,
"proximity": 1,
"positions": 0,
"term_frequencies": 1,
"offsets": 0,
"payloads": 0
},
"stored_fields": 2,
"doc_values": 1,
"points": 0,
"norms": 1,
"term_vectors": 0,
"knn_vectors": 0
},
"fields": {
"_id": {
"any": 1,
"inverted_index": {
"terms": 1,
"postings": 1,
"proximity": 1,
"positions": 0,
"term_frequencies": 1,
"offsets": 0,
"payloads": 0
},
"stored_fields": 1,
"doc_values": 0,
"points": 0,
"norms": 0,
"term_vectors": 0,
"knn_vectors": 0
},
"_source": {},
"context": {},
"message.keyword": {}
}
}
}
]
}
}All methods and paths for this operation:
Flushing a data stream or index is the process of making sure that any data that is currently only stored in the transaction log is also permanently stored in the Lucene index. When restarting, Elasticsearch replays any unflushed operations from the transaction log into the Lucene index to bring it back into the state that it was in before the restart. Elasticsearch automatically triggers flushes as needed, using heuristics that trade off the size of the unflushed transaction log against the cost of performing each flush.
After each operation has been flushed it is permanently stored in the Lucene index. This may mean that there is no need to maintain an additional copy of it in the transaction log. The transaction log is made up of multiple files, called generations, and Elasticsearch will delete any generation files when they are no longer needed, freeing up disk space.
It is also possible to trigger a flush on one or more indices using the flush API, although it is rare for users to need to call this API directly. If you call the flush API after indexing some documents then a successful response indicates that Elasticsearch has flushed all the documents that were indexed before the flush API was called.
maintenanceComma-separated list of data streams, indices, and aliases to flush.
Supports wildcards (*).
To flush all data streams and indices, omit this parameter or use * or _all.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, the request forces a flush even if there are no changes to commit to the index.
If true, the flush operation blocks until execution when another flush operation is running.
If false, Elasticsearch returns an error if you request a flush when another flush operation is running.
POST /_flush
resp = client.indices.flush()const response = await client.indices.flush();response = client.indices.flush$resp = $client->indices()->flush();curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_flush"client.indices().flush(f -> f);
All methods and paths for this operation:
Perform the force merge operation on the shards of one or more indices. For data streams, the API forces a merge on the shards of the stream's backing indices.
Merging reduces the number of segments in each shard by merging some of them together and also frees up the space used by deleted documents. Merging normally happens automatically, but sometimes it is useful to trigger a merge manually.
WARNING: We recommend force merging only a read-only index (meaning the index is no longer receiving writes). When documents are updated or deleted, the old version is not immediately removed but instead soft-deleted and marked with a "tombstone". These soft-deleted documents are automatically cleaned up during regular segment merges. But force merge can cause very large (greater than 5 GB) segments to be produced, which are not eligible for regular merges. So the number of soft-deleted documents can then grow rapidly, resulting in higher disk usage and worse search performance. If you regularly force merge an index receiving writes, this can also make snapshots more expensive, since the new documents can't be backed up incrementally.
Blocks during a force merge
Calls to this API block until the merge is complete (unless request contains wait_for_completion=false).
If the client connection is lost before completion then the force merge process will continue in the background.
Any new requests to force merge the same indices will also block until the ongoing force merge is complete.
Running force merge asynchronously
If the request contains wait_for_completion=false, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to get the status of the task.
However, you can not cancel this task as the force merge task is not cancelable.
Elasticsearch creates a record of this task as a document at _tasks/<task_id>.
When you are done with a task, you should delete the task document so Elasticsearch can reclaim the space.
Force merging multiple indices
You can force merge multiple indices with a single request by targeting:
Each targeted shard is force-merged separately using the force_merge threadpool.
By default each node only has a single force_merge thread which means that the shards on that node are force-merged one at a time.
If you expand the force_merge threadpool on a node then it will force merge its shards in parallel
Force merge makes the storage for the shard being merged temporarily increase, as it may require free space up to triple its size in case max_num_segments parameter is set to 1, to rewrite all segments into a new one.
Data streams and time-based indices
Force-merging is useful for managing a data stream's older backing indices and other time-based indices, particularly after a rollover. In these cases, each index only receives indexing traffic for a certain period of time. Once an index receive no more writes, its shards can be force-merged to a single segment. This can be a good idea because single-segment shards can sometimes use simpler and more efficient data structures to perform searches. For example:
POST /.ds-my-data-stream-2099.03.07-000001/_forcemerge?max_num_segments=1
maintenanceA comma-separated list of index names; use _all or empty string to perform the operation on all indices
Whether to ignore if a wildcard indices expression resolves into no concrete indices. (This includes _all string or when no indices have been specified)
Whether to expand wildcard expression to concrete indices that are open, closed or both.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Specify whether the index should be flushed after performing the operation (default: true)
The number of segments the index should be merged into (default: dynamic)
Specify whether the operation should only expunge deleted documents
Should the request wait until the force merge is completed.
POST my-index-000001/_forcemerge
resp = client.indices.forcemerge(
index="my-index-000001",
)const response = await client.indices.forcemerge({
index: "my-index-000001",
});response = client.indices.forcemerge(
index: "my-index-000001"
)$resp = $client->indices()->forcemerge([
"index" => "my-index-000001",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_forcemerge"client.indices().forcemerge(f -> f
.index("my-index-000001")
);
For data streams, the API opens any closed backing indices.
A closed index is blocked for read/write operations and does not allow all operations that opened indices allow. It is not possible to index documents or to search for documents in a closed index. This allows closed indices to not have to maintain internal data structures for indexing or searching documents, resulting in a smaller overhead on the cluster.
When opening or closing an index, the master is responsible for restarting the index shards to reflect the new state of the index. The shards will then go through the normal recovery process. The data of opened or closed indices is automatically replicated by the cluster to ensure that enough shard copies are safely kept around at all times.
You can open and close multiple indices.
An error is thrown if the request explicitly refers to a missing index.
This behavior can be turned off by using the ignore_unavailable=true parameter.
By default, you must explicitly name the indices you are opening or closing.
To open or close indices with _all, *, or other wildcard expressions, change the action.destructive_requires_name setting to false.
This setting can also be changed with the cluster update settings API.
Closed indices consume a significant amount of disk-space which can cause problems in managed environments.
Closing indices can be turned off with the cluster settings API by setting cluster.indices.close.enable to false.
Because opening or closing an index allocates its shards, the wait_for_active_shards setting on index creation applies to the _open and _close index actions as well.
manageComma-separated list of data streams, indices, and aliases used to limit the request.
Supports wildcards (*).
By default, you must explicitly name the indices you using to limit the request.
To limit a request using _all, *, or other wildcard expressions, change the action.destructive_requires_name setting to false.
You can update this setting in the elasticsearch.yml file or using the cluster update settings API.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
The number of shard copies that must be active before proceeding with the operation.
Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
Values are all or index-setting.
POST /.ds-my-data-stream-2099.03.07-000001/_open/
resp = client.indices.open(
index=".ds-my-data-stream-2099.03.07-000001",
)const response = await client.indices.open({
index: ".ds-my-data-stream-2099.03.07-000001",
});response = client.indices.open(
index: ".ds-my-data-stream-2099.03.07-000001"
)$resp = $client->indices()->open([
"index" => ".ds-my-data-stream-2099.03.07-000001",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/.ds-my-data-stream-2099.03.07-000001/_open/"client.indices().open(o -> o
.index(".ds-my-data-stream-2099.03.07-000001")
);
{
"acknowledged" : true,
"shards_acknowledged" : true
}All methods and paths for this operation:
Changes dynamic index settings in real time. For data streams, index setting changes are applied to all backing indices by default.
To revert a setting to the default value, use a null value.
The list of per-index settings that can be updated dynamically on live indices can be found in index settings documentation.
To preserve existing settings from being updated, set the preserve_existing parameter to true.
There are multiple valid ways to represent index settings in the request body. You can specify only the setting, for example:
{
"number_of_replicas": 1
}
Or you can use an index setting object:
{
"index": {
"number_of_replicas": 1
}
}
Or you can use dot annotation:
{
"index.number_of_replicas": 1
}
Or you can embed any of the aforementioned options in a settings object. For example:
{
"settings": {
"index": {
"number_of_replicas": 1
}
}
}
NOTE: You can only define new analyzers on closed indices. To add an analyzer, you must close the index, define the analyzer, and reopen the index. You cannot close the write index of a data stream. To update the analyzer for a data stream's write index and future backing indices, update the analyzer in the index template used by the stream. Then roll over the data stream to apply the new analyzer to the stream's write index and future backing indices. This affects searches and any new data added to the stream after the rollover. However, it does not affect the data stream's backing indices or their existing data. To change the analyzer for existing backing indices, you must create a new data stream and reindex your data into it.
manageComma-separated list of data streams, indices, and aliases used to limit
the request. Supports wildcards (*). To target all data streams and
indices, omit this parameter or use * or _all.
If false, the request returns an error if any wildcard expression, index
alias, or _all value targets only missing or closed indices. This
behavior applies even if the request targets other open indices. For
example, a request targeting foo*,bar* returns an error if an index
starts with foo but no index starts with bar.
Type of index that wildcard patterns can match. If the request can target
data streams, this argument determines whether wildcard expressions match
hidden data streams. Supports comma-separated values, such as
open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, returns settings in flat format.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
If true, existing index settings remain unchanged.
Whether to close and reopen the index to apply non-dynamic settings.
If set to true the indices to which the settings are being applied
will be closed temporarily and then reopened in order to apply the changes.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
PUT /my-index-000001/_settings
{
"index" : {
"number_of_replicas" : 2
}
}resp = client.indices.put_settings(
index="my-index-000001",
settings={
"index": {
"number_of_replicas": 2
}
},
)const response = await client.indices.putSettings({
index: "my-index-000001",
settings: {
index: {
number_of_replicas: 2,
},
},
});response = client.indices.put_settings(
index: "my-index-000001",
body: {
"index": {
"number_of_replicas": 2
}
}
)$resp = $client->indices()->putSettings([
"index" => "my-index-000001",
"body" => [
"index" => [
"number_of_replicas" => 2,
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"index":{"number_of_replicas":2}}' "$ELASTICSEARCH_URL/my-index-000001/_settings"client.indices().putSettings(p -> p
.index("my-index-000001")
.settings(s -> s
.index(i -> i
.numberOfReplicas("2")
)
)
);
{
"index" : {
"number_of_replicas" : 2
}
}{
"index" : {
"refresh_interval" : null
}
}{
"analysis": {
"analyzer": {
"content": {
"type": "custom",
"tokenizer": "whitespace"
}
}
}
}All methods and paths for this operation:
A refresh makes recent operations performed on one or more indices available for search. For data streams, the API runs the refresh operation on the stream’s backing indices.
By default, Elasticsearch periodically refreshes indices every second, but only on indices that have received one search request or more in the last 30 seconds.
You can change this default interval with the index.refresh_interval setting.
Refresh requests are synchronous and do not return a response until the refresh operation completes.
Refreshes are resource-intensive. To ensure good cluster performance, it's recommended to wait for Elasticsearch's periodic refresh rather than performing an explicit refresh when possible.
If your application workflow indexes documents and then runs a search to retrieve the indexed document, it's recommended to use the index API's refresh=wait_for query parameter option.
This option ensures the indexing operation waits for a periodic refresh before running the search.
maintenanceComma-separated list of data streams, indices, and aliases used to limit the request.
Supports wildcards (*).
To target all data streams and indices, omit this parameter or use * or _all.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
GET _refresh
resp = client.indices.refresh()const response = await client.indices.refresh();response = client.indices.refresh$resp = $client->indices()->refresh();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_refresh"client.indices().refresh(r -> r);
All methods and paths for this operation:
Resolve the specified index expressions to return information about each cluster, including the local "querying" cluster, if included. If no index expression is provided, the API will return information about all the remote clusters that are configured on the querying cluster.
This endpoint is useful before doing a cross-cluster search in order to determine which remote clusters should be included in a search.
You use the same index expression with this endpoint as you would for cross-cluster search. Index and cluster exclusions are also supported with this endpoint.
For each cluster in the index expression, information is returned about:
remote/info endpoint.skip_unavailable as true or false.For example, GET /_resolve/cluster/my-index-*,cluster*:my-index-* returns information about the local cluster and all remotely configured clusters that start with the alias cluster*.
Each cluster returns information about whether it has any indices, aliases or data streams that match my-index-*.
The ability to query without an index expression was added in version 8.18, so when
querying remote clusters older than that, the local cluster will send the index
expression dummy* to those remote clusters. Thus, if an errors occur, you may see a reference
to that index expression even though you didn't request it. If it causes a problem, you can
instead include an index expression like *:* to bypass the issue.
You may want to exclude a cluster or index from a search when:
skip_unavailable=false. Running a cross-cluster search under those conditions will cause the entire search to fail.logs*,remote1:logs* and the remote1 cluster has no indices, aliases or data streams that match logs*. In that case, that cluster will return no results from that cluster if you include it in a cross-cluster search._resolve/cluster response will be present. (This is also where security/permission errors will be shown.)The remote/info endpoint is commonly used to test whether the "local" cluster (the cluster being queried) is connected to its remote clusters, but it does not necessarily reflect whether the remote cluster is available or not.
The remote cluster may be available, while the local cluster is not currently connected to it.
You can use the _resolve/cluster API to attempt to reconnect to remote clusters.
For example with GET _resolve/cluster or GET _resolve/cluster/*:*.
The connected field in the response will indicate whether it was successful.
If a connection was (re-)established, this will also cause the remote/info endpoint to now indicate a connected status.
view_index_metadataA comma-separated list of names or index patterns for the indices, aliases, and data streams to resolve.
Resources on remote clusters can be specified using the <cluster>:<name> syntax.
Index and cluster exclusions (e.g., -cluster1:*) are also supported.
If no index expression is specified, information about all remote clusters configured on the local cluster
is returned without doing any index matching
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing
or closed indices. This behavior applies even if the request targets other open indices. For example, a request
targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.
NOTE: This option is only supported when specifying an index expression. You will get an error if you specify index
options to the _resolve/cluster API endpoint that takes no index expression.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
NOTE: This option is only supported when specifying an index expression. You will get an error if you specify index
options to the _resolve/cluster API endpoint that takes no index expression.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, concrete, expanded, or aliased indices are ignored when frozen.
NOTE: This option is only supported when specifying an index expression. You will get an error if you specify index
options to the _resolve/cluster API endpoint that takes no index expression.
The maximum time to wait for remote clusters to respond. If a remote cluster does not respond within this timeout period, the API response will show the cluster as not connected and include an error message that the request timed out.
The default timeout is unset and the query can take as long as the networking layer is configured to wait for remote clusters that are not responding (typically 30 seconds).
Values are -1 or 0.
GET /_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailable=false&timeout=5s
resp = client.indices.resolve_cluster(
name="not-present,clust*:my-index*,oldcluster:*",
ignore_unavailable=False,
timeout="5s",
)const response = await client.indices.resolveCluster({
name: "not-present,clust*:my-index*,oldcluster:*",
ignore_unavailable: "false",
timeout: "5s",
});response = client.indices.resolve_cluster(
name: "not-present,clust*:my-index*,oldcluster:*",
ignore_unavailable: "false",
timeout: "5s"
)$resp = $client->indices()->resolveCluster([
"name" => "not-present,clust*:my-index*,oldcluster:*",
"ignore_unavailable" => "false",
"timeout" => "5s",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_resolve/cluster/not-present,clust*:my-index*,oldcluster:*?ignore_unavailable=false&timeout=5s"client.indices().resolveCluster(r -> r
.ignoreUnavailable(false)
.name(List.of("not-present","clust*:my-index*","oldcluster:*"))
.timeout(t -> t
.offset(5)
)
);
{
"(local)": {
"connected": true,
"skip_unavailable": false,
"matching_indices": true,
"version": {
"number": "8.13.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
},
"cluster_one": {
"connected": true,
"skip_unavailable": true,
"matching_indices": true,
"version": {
"number": "8.13.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
},
"cluster_two": {
"connected": true,
"skip_unavailable": false,
"matching_indices": true,
"version": {
"number": "8.13.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
}
}{
"(local)": {
"connected": true,
"skip_unavailable": false,
"error": "no such index [not_present]"
},
"cluster_one": {
"connected": true,
"skip_unavailable": true,
"matching_indices": false,
"version": {
"number": "8.13.0",
"build_flavor": "default",
"minimum_wire_compatibility_version": "7.17.0",
"minimum_index_compatibility_version": "7.0.0"
}
},
"cluster_two": {
"connected": false,
"skip_unavailable": false
},
"cluster_three": {
"connected": false,
"skip_unavailable": false,
"error": "Request timed out before receiving a response from the remote cluster"
},
"oldcluster": {
"connected": true,
"skip_unavailable": false,
"matching_indices": true
}
}Comma-separated name(s) or index pattern(s) of the indices, aliases, and data streams to resolve.
Resources on remote clusters can be specified using the <cluster>:<name> syntax.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.
GET /_resolve/index/f*,remoteCluster1:bar*?expand_wildcards=all
resp = client.indices.resolve_index(
name="f*,remoteCluster1:bar*",
expand_wildcards="all",
)const response = await client.indices.resolveIndex({
name: "f*,remoteCluster1:bar*",
expand_wildcards: "all",
});response = client.indices.resolve_index(
name: "f*,remoteCluster1:bar*",
expand_wildcards: "all"
)$resp = $client->indices()->resolveIndex([
"name" => "f*,remoteCluster1:bar*",
"expand_wildcards" => "all",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_resolve/index/f*,remoteCluster1:bar*?expand_wildcards=all"{
"indices": [
{
"name": "foo_closed",
"attributes": [
"closed"
]
},
{
"name": "freeze-index",
"aliases": [
"f-alias"
],
"attributes": [
"open"
]
},
{
"name": "remoteCluster1:bar-01",
"attributes": [
"open"
]
}
],
"aliases": [
{
"name": "f-alias",
"indices": [
"freeze-index",
"my-index-000001"
]
}
],
"data_streams": [
{
"name": "foo",
"backing_indices": [
".ds-foo-2099.03.07-000001"
],
"timestamp_field": "@timestamp"
}
]
}All methods and paths for this operation:
TIP: It is recommended to use the index lifecycle rollover action to automate rollovers.
The rollover API creates a new index for a data stream or index alias. The API behavior depends on the rollover target.
Roll over a data stream
If you roll over a data stream, the API creates a new write index for the stream. The stream's previous write index becomes a regular backing index. A rollover also increments the data stream's generation.
Roll over an index alias with a write index
TIP: Prior to Elasticsearch 7.9, you'd typically use an index alias with a write index to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with data tiers.
If an index alias points to multiple indices, one of the indices must be a write index.
The rollover API creates a new write index for the alias with is_write_index set to true.
The API also sets is_write_index to false for the previous write index.
Roll over an index alias with one index
If you roll over an index alias that points to only one index, the API creates a new index for the alias and removes the original index from the alias.
NOTE: A rollover creates a new index and is subject to the wait_for_active_shards setting.
Increment index names for an alias
When you roll over an index alias, you can specify a name for the new index.
If you don't specify a name and the current index ends with - and a number, such as my-index-000001 or my-index-3, the new index name increments that number.
For example, if you roll over an alias with a current index of my-index-000001, the rollover creates a new index named my-index-000002.
This number is always six characters and zero-padded, regardless of the previous index's name.
If you use an index alias for time series data, you can use date math in the index name to track the rollover date.
For example, you can create an alias that points to an index named <my-index-{now/d}-000001>.
If you create the index on May 6, 2099, the index's name is my-index-2099.05.06-000001.
If you roll over the alias on May 7, 2099, the new index's name is my-index-2099.05.07-000002.
manageName of the data stream or index alias to roll over.
Name of the index to create. Supports date math. Data streams do not support this parameter.
If true, checks whether the current index satisfies the specified conditions but does not perform a rollover.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
The number of shard copies that must be active before proceeding with the operation.
Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
Values are all or index-setting.
If set to true, the rollover action will only mark a data stream to signal that it needs to be rolled over at the next write. Only allowed on data streams.
POST my-data-stream/_rollover
{
"conditions": {
"max_age": "7d",
"max_docs": 1000,
"max_primary_shard_size": "50gb",
"max_primary_shard_docs": "2000"
}
}resp = client.indices.rollover(
alias="my-data-stream",
conditions={
"max_age": "7d",
"max_docs": 1000,
"max_primary_shard_size": "50gb",
"max_primary_shard_docs": "2000"
},
)const response = await client.indices.rollover({
alias: "my-data-stream",
conditions: {
max_age: "7d",
max_docs: 1000,
max_primary_shard_size: "50gb",
max_primary_shard_docs: "2000",
},
});response = client.indices.rollover(
alias: "my-data-stream",
body: {
"conditions": {
"max_age": "7d",
"max_docs": 1000,
"max_primary_shard_size": "50gb",
"max_primary_shard_docs": "2000"
}
}
)$resp = $client->indices()->rollover([
"alias" => "my-data-stream",
"body" => [
"conditions" => [
"max_age" => "7d",
"max_docs" => 1000,
"max_primary_shard_size" => "50gb",
"max_primary_shard_docs" => "2000",
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"conditions":{"max_age":"7d","max_docs":1000,"max_primary_shard_size":"50gb","max_primary_shard_docs":"2000"}}' "$ELASTICSEARCH_URL/my-data-stream/_rollover"client.indices().rollover(r -> r
.alias("my-data-stream")
.conditions(c -> c
.maxAge(m -> m
.time("7d")
)
.maxDocs(1000L)
.maxPrimaryShardSize("50gb")
.maxPrimaryShardDocs(2000L)
)
);
{
"conditions": {
"max_age": "7d",
"max_docs": 1000,
"max_primary_shard_size": "50gb",
"max_primary_shard_docs": "2000"
}
}{
"_shards": {},
"indices": {
"test": {
"shards": {
"0": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "zDC_RorJQCao9xf9pg3Fvw"
},
"num_committed_segments": 0,
"num_search_segments": 1,
"segments": {
"_0": {
"generation": 0,
"num_docs": 1,
"deleted_docs": 0,
"size_in_bytes": 3800,
"committed": false,
"search": true,
"version": "7.0.0",
"compound": true,
"attributes": {}
}
}
}
]
}
}
}
}Comma-separated list of data streams, indices, and aliases used to limit the request.
Supports wildcards (*).
To target all data streams and indices, omit this parameter or use * or _all.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, the request returns a verbose response.
GET /my-index-000001/_segments
resp = client.indices.segments(
index="my-index-000001",
)const response = await client.indices.segments({
index: "my-index-000001",
});response = client.indices.segments(
index: "my-index-000001"
)$resp = $client->indices()->segments([
"index" => "my-index-000001",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_segments"client.indices().segments(s -> s
.index("my-index-000001")
);
{
"acknowledged": true,
"shards_acknowledged": true,
"old_index": ".ds-my-data-stream-2099.05.06-000001",
"new_index": ".ds-my-data-stream-2099.05.07-000002",
"rolled_over": true,
"dry_run": false,
"lazy": false,
"conditions": {
"[max_age: 7d]": false,
"[max_docs: 1000]": true,
"[max_primary_shard_size: 50gb]": false,
"[max_primary_shard_docs: 2000]": false
}
}Whether the index template we optionally defined in the body should only be dry-run added if new or can also replace an existing one
User defined reason for dry-run creating the new template for simulation purposes
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
If true, returns all relevant default configurations for the index template.
POST /_index_template/_simulate_index/my-index-000001
resp = client.indices.simulate_index_template(
name="my-index-000001",
)const response = await client.indices.simulateIndexTemplate({
name: "my-index-000001",
});response = client.indices.simulate_index_template(
name: "my-index-000001"
)$resp = $client->indices()->simulateIndexTemplate([
"name" => "my-index-000001",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_index_template/_simulate_index/my-index-000001"client.indices().simulateIndexTemplate(s -> s
.name("my-index-000001")
);
{
"template" : {
"settings" : {
"index" : {
"number_of_shards" : "2",
"number_of_replicas" : "0",
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
}
}
},
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
}
}
},
"aliases" : { }
},
"overlapping" : [
{
"name" : "template_1",
"index_patterns" : [
"my-index-*"
]
}
]
}All methods and paths for this operation:
Validates a query without running it.
Comma-separated list of data streams, indices, and aliases to search.
Supports wildcards (*).
To search all data streams or indices, omit this parameter or use * or _all.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
If true, the validation is executed on all shards instead of one random shard per index.
Analyzer to use for the query string.
This parameter can only be used when the q query string parameter is specified.
If true, wildcard and prefix queries are analyzed.
The default operator for query string query: AND or OR.
Values are and, AND, or, or OR.
Field to use as default where no field prefix is given in the query string.
This parameter can only be used when the q query string parameter is specified.
Type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, the response returns detailed information if an error has occurred.
If true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
If true, returns a more detailed explanation showing the actual Lucene query that will be executed.
Query in the Lucene query string syntax.
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
GET my-index-000001/_validate/query?q=user.id:kimchy
resp = client.indices.validate_query(
index="my-index-000001",
q="user.id:kimchy",
)const response = await client.indices.validateQuery({
index: "my-index-000001",
q: "user.id:kimchy",
});response = client.indices.validate_query(
index: "my-index-000001",
q: "user.id:kimchy"
)$resp = $client->indices()->validateQuery([
"index" => "my-index-000001",
"q" => "user.id:kimchy",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_validate/query?q=user.id:kimchy"client.indices().validateQuery(v -> v
.index("my-index-000001")
.q("user.id:kimchy")
);
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
GET _ilm/policy/my_policy
resp = client.ilm.get_lifecycle(
name="my_policy",
)const response = await client.ilm.getLifecycle({
name: "my_policy",
});response = client.ilm.get_lifecycle(
policy: "my_policy"
)$resp = $client->ilm()->getLifecycle([
"policy" => "my_policy",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_ilm/policy/my_policy"client.ilm().getLifecycle(g -> g
.name("my_policy")
);
{
"my_policy": {
"version": 1,
"modified_date": 82392349,
"policy": {
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
},
"in_use_by" : {
"indices" : [],
"data_streams" : [],
"composable_templates" : []
}
}
}If the specified policy exists, it is replaced and the policy version is incremented.
NOTE: Only the latest version of the policy is stored, you cannot revert to previous versions.
managemanage_ilmPeriod to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
PUT _ilm/policy/my_policy
{
"policy": {
"_meta": {
"description": "used for nginx log",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}resp = client.ilm.put_lifecycle(
name="my_policy",
policy={
"_meta": {
"description": "used for nginx log",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
},
)const response = await client.ilm.putLifecycle({
name: "my_policy",
policy: {
_meta: {
description: "used for nginx log",
project: {
name: "myProject",
department: "myDepartment",
},
},
phases: {
warm: {
min_age: "10d",
actions: {
forcemerge: {
max_num_segments: 1,
},
},
},
delete: {
min_age: "30d",
actions: {
delete: {},
},
},
},
},
});response = client.ilm.put_lifecycle(
policy: "my_policy",
body: {
"policy": {
"_meta": {
"description": "used for nginx log",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}
)$resp = $client->ilm()->putLifecycle([
"policy" => "my_policy",
"body" => [
"policy" => [
"_meta" => [
"description" => "used for nginx log",
"project" => [
"name" => "myProject",
"department" => "myDepartment",
],
],
"phases" => [
"warm" => [
"min_age" => "10d",
"actions" => [
"forcemerge" => [
"max_num_segments" => 1,
],
],
],
"delete" => [
"min_age" => "30d",
"actions" => [
"delete" => new ArrayObject([]),
],
],
],
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"policy":{"_meta":{"description":"used for nginx log","project":{"name":"myProject","department":"myDepartment"}},"phases":{"warm":{"min_age":"10d","actions":{"forcemerge":{"max_num_segments":1}}},"delete":{"min_age":"30d","actions":{"delete":{}}}}}}' "$ELASTICSEARCH_URL/_ilm/policy/my_policy"client.ilm().putLifecycle(p -> p
.name("my_policy")
.policy(po -> po
.phases(ph -> ph
.delete(d -> d
.actions(a -> a
.delete(de -> de)
)
.minAge(m -> m
.time("30d")
)
)
.warm(w -> w
.actions(a -> a
.forcemerge(f -> f
.maxNumSegments(1)
)
)
.minAge(m -> m
.time("10d")
)
)
)
.meta(Map.of("description", JsonData.fromJson("\"used for nginx log\""),"project", JsonData.fromJson("{\"name\":\"myProject\",\"department\":\"myDepartment\"}")))
)
);
{
"policy": {
"_meta": {
"description": "used for nginx log",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}{
"acknowledged": true
}Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
DELETE _ilm/policy/my_policy
resp = client.ilm.delete_lifecycle(
name="my_policy",
)const response = await client.ilm.deleteLifecycle({
name: "my_policy",
});response = client.ilm.delete_lifecycle(
policy: "my_policy"
)$resp = $client->ilm()->deleteLifecycle([
"policy" => "my_policy",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_ilm/policy/my_policy"client.ilm().deleteLifecycle(d -> d
.name("my_policy")
);
{
"acknowledged": true
}Get the current lifecycle status for one or more indices. For data streams, the API retrieves the current lifecycle status for the stream's backing indices.
The response indicates when the index entered each lifecycle state, provides the definition of the running phase, and information about any failures.
view_index_metadata,manage_ilmComma-separated list of data streams, indices, and aliases to target. Supports wildcards (*).
To target all data streams and indices, use * or _all.
Filters the returned indices to only indices that are managed by ILM and are in an error state, either due to an encountering an error while executing the policy, or attempting to use a policy that does not exist.
Filters the returned indices to only indices that are managed by ILM.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
GET .ds-timeseries-*/_ilm/explain
resp = client.ilm.explain_lifecycle(
index=".ds-timeseries-*",
)const response = await client.ilm.explainLifecycle({
index: ".ds-timeseries-*",
});response = client.ilm.explain_lifecycle(
index: ".ds-timeseries-*"
)$resp = $client->ilm()->explainLifecycle([
"index" => ".ds-timeseries-*",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/.ds-timeseries-*/_ilm/explain"client.ilm().explainLifecycle(e -> e
.index(".ds-timeseries-*")
);
{
"indices": {
"my-index-000001": {
"index": "my-index-000001",
"index_creation_date_millis": 1538475653281,
"index_creation_date": "2018-10-15T13:45:21.981Z",
"time_since_index_creation": "15s",
"managed": true,
"policy": "my_policy",
"lifecycle_date_millis": 1538475653281,
"lifecycle_date": "2018-10-15T13:45:21.981Z",
"age": "15s",
"phase": "new",
"phase_time_millis": 1538475653317,
"phase_time": "2018-10-15T13:45:22.577Z",
"action": "complete"
"action_time_millis": 1538475653317,
"action_time": "2018-10-15T13:45:22.577Z",
"step": "complete",
"step_time_millis": 1538475653317,
"step_time": "2018-10-15T13:45:22.577Z"
}
}
}POST logs-my_app-default/_ilm/remove
resp = client.ilm.remove_policy(
index="logs-my_app-default",
)const response = await client.ilm.removePolicy({
index: "logs-my_app-default",
});response = client.ilm.remove_policy(
index: "logs-my_app-default"
)$resp = $client->ilm()->removePolicy([
"index" => "logs-my_app-default",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/logs-my_app-default/_ilm/remove"client.ilm().removePolicy(r -> r
.index("logs-my_app-default")
);
{
"has_failures" : false,
"failed_indexes" : []
}Halt all lifecycle management operations and stop the index lifecycle management plugin. This is useful when you are performing maintenance on the cluster and need to prevent ILM from performing any actions on your indices.
The API returns as soon as the stop request has been acknowledged, but the plugin might continue to run until in-progress operations complete and the plugin can be safely stopped. Use the get ILM status API to check whether ILM is running.
manage_ilmPeriod to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
POST _ilm/stop
resp = client.ilm.stop()const response = await client.ilm.stop();response = client.ilm.stop$resp = $client->ilm()->stop();curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_ilm/stop"client.ilm().stop(s -> s);
{
"acknowledged": true
}Inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.
The chat completion inference API enables real-time responses for chat completion tasks by delivering answers incrementally, reducing response times during computation.
It only works with the chat_completion task type for openai and elastic inference services.
NOTE: The chat_completion task type is only available within the _stream API and only supports streaming.
The Chat completion inference API and the Stream inference API differ in their response structure and capabilities.
The Chat completion inference API provides more comprehensive customization options through more fields and function calling support.
If you use the openai, hugging_face or the elastic service, use the Chat completion inference API.
Specifies the amount of time to wait for the inference request to complete.
Values are -1 or 0.
A list of objects representing the conversation.
Requests should generally only add new messages from the user (role user).
The other message roles (assistant, system, or tool) should generally only be copied from the response to a previous completion request, such that the messages array is built up throughout a conversation.
An object representing part of the conversation.
The ID of the model to use.
The upper bound limit for the number of tokens that can be generated for a completion request.
A sequence of strings to control when the model should stop generating additional tokens.
The sampling temperature to use.
A list of tools that the model can call. Example:
{
"tools": [
{
"type": "function",
"function": {
"name": "get_price_of_item",
"description": "Get the current price of an item",
"parameters": {
"type": "object",
"properties": {
"item": {
"id": "12345"
},
"unit": {
"type": "currency"
}
}
}
}
}
]
}
A list of tools that the model can call.
Nucleus sampling, an alternative to sampling with temperature.
POST _inference/chat_completion/openai-completion/_stream
{
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "What is Elastic?"
}
]
}resp = client.inference.chat_completion_unified(
inference_id="openai-completion",
chat_completion_request={
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "What is Elastic?"
}
]
},
)const response = await client.inference.chatCompletionUnified({
inference_id: "openai-completion",
chat_completion_request: {
model: "gpt-4o",
messages: [
{
role: "user",
content: "What is Elastic?",
},
],
},
});response = client.inference.chat_completion_unified(
inference_id: "openai-completion",
body: {
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "What is Elastic?"
}
]
}
)$resp = $client->inference()->chatCompletionUnified([
"inference_id" => "openai-completion",
"body" => [
"model" => "gpt-4o",
"messages" => array(
[
"role" => "user",
"content" => "What is Elastic?",
],
),
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"model":"gpt-4o","messages":[{"role":"user","content":"What is Elastic?"}]}' "$ELASTICSEARCH_URL/_inference/chat_completion/openai-completion/_stream"client.inference().chatCompletionUnified(c -> c
.inferenceId("openai-completion")
.chatCompletionRequest(ch -> ch
.messages(m -> m
.content(co -> co
.string("What is Elastic?")
)
.role("user")
)
.model("gpt-4o")
)
);
{
"model": "gpt-4o",
"messages": [
{
"role": "user",
"content": "What is Elastic?"
}
]
}{
"messages": [
{
"role": "assistant",
"content": "Let's find out what the weather is",
"tool_calls": [
{
"id": "call_KcAjWtAww20AihPHphUh46Gd",
"type": "function",
"function": {
"name": "get_current_weather",
"arguments": "{\"location\":\"Boston, MA\"}"
}
}
]
},
{
"role": "tool",
"content": "The weather is cold",
"tool_call_id": "call_KcAjWtAww20AihPHphUh46Gd"
}
]
}{
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's the price of a scarf?"
}
]
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_current_price",
"description": "Get the current price of a item",
"parameters": {
"type": "object",
"properties": {
"item": {
"id": "123"
}
}
}
}
}
],
"tool_choice": {
"type": "function",
"function": {
"name": "get_current_price"
}
}
}event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":"","role":"assistant"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}
event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":Elastic"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}
event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[{"delta":{"content":" is"},"index":0}],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk"}}
(...)
event: message
data: {"chat_completion":{"id":"chatcmpl-Ae0TWsy2VPnSfBbv5UztnSdYUMFP3","choices":[],"model":"gpt-4o-2024-08-06","object":"chat.completion.chunk","usage":{"completion_tokens":28,"prompt_tokens":16,"total_tokens":44}}}
event: message
data: [DONE]Specifies the amount of time to wait for the inference request to complete.
Values are -1 or 0.
POST _inference/completion/openai_chat_completions
{
"input": "What is Elastic?"
}resp = client.inference.completion(
inference_id="openai_chat_completions",
input="What is Elastic?",
)const response = await client.inference.completion({
inference_id: "openai_chat_completions",
input: "What is Elastic?",
});response = client.inference.completion(
inference_id: "openai_chat_completions",
body: {
"input": "What is Elastic?"
}
)$resp = $client->inference()->completion([
"inference_id" => "openai_chat_completions",
"body" => [
"input" => "What is Elastic?",
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"input":"What is Elastic?"}' "$ELASTICSEARCH_URL/_inference/completion/openai_chat_completions"client.inference().completion(c -> c
.inferenceId("openai_chat_completions")
.input("What is Elastic?")
);
{
"input": "What is Elastic?"
}{
"completion": [
{
"result": "Elastic is a company that provides a range of software solutions for search, logging, security, and analytics. Their flagship product is Elasticsearch, an open-source, distributed search engine that allows users to search, analyze, and visualize large volumes of data in real-time. Elastic also offers products such as Kibana, a data visualization tool, and Logstash, a log management and pipeline tool, as well as various other tools and solutions for data analysis and management."
}
]
}All methods and paths for this operation:
GET _inference/sparse_embedding/my-elser-model
resp = client.inference.get(
task_type="sparse_embedding",
inference_id="my-elser-model",
)const response = await client.inference.get({
task_type: "sparse_embedding",
inference_id: "my-elser-model",
});response = client.inference.get(
task_type: "sparse_embedding",
inference_id: "my-elser-model"
)$resp = $client->inference()->get([
"task_type" => "sparse_embedding",
"inference_id" => "my-elser-model",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_inference/sparse_embedding/my-elser-model"client.inference().get(g -> g
.inferenceId("my-elser-model")
.taskType(TaskType.SparseEmbedding)
);
All methods and paths for this operation:
IMPORTANT: The inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Mistral, Azure OpenAI, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.
The following integrations are available through the inference API. You can find the available task types next to the integration name:
completion, rerank, sparse_embedding, text_embedding)completion, text_embedding)chat_completion, completion, rerank, sparse_embedding, text_embedding)completion)completion, text_embedding)completion, text_embedding)completion, rerank, text_embedding)chat_completion, completion)rerank, sparse_embedding, text_embedding - this service is for built-in models and models uploaded through Eland)sparse_embedding)completion, text_embedding)chat_completion, completion, rerank, text_embedding)chat_completion, completion, rerank, text_embedding)rerank, text_embedding)chat_completion, completion, text_embedding)chat_completion, completion, text_embedding)chat_completion, completion, text_embedding)rerank, text_embedding)text_embedding)manage_inferenceThe task type. Refer to the integration list in the API description for the available task types.
Values are sparse_embedding, text_embedding, rerank, completion, or chat_completion.
The inference Id
Specifies the amount of time to wait for the inference endpoint to be created.
Values are -1 or 0.
PUT _inference/rerank/my-rerank-model
{
"service": "cohere",
"service_settings": {
"model_id": "rerank-english-v3.0",
"api_key": "{{COHERE_API_KEY}}"
}
}resp = client.inference.put(
task_type="rerank",
inference_id="my-rerank-model",
inference_config={
"service": "cohere",
"service_settings": {
"model_id": "rerank-english-v3.0",
"api_key": "{{COHERE_API_KEY}}"
}
},
)const response = await client.inference.put({
task_type: "rerank",
inference_id: "my-rerank-model",
inference_config: {
service: "cohere",
service_settings: {
model_id: "rerank-english-v3.0",
api_key: "{{COHERE_API_KEY}}",
},
},
});response = client.inference.put(
task_type: "rerank",
inference_id: "my-rerank-model",
body: {
"service": "cohere",
"service_settings": {
"model_id": "rerank-english-v3.0",
"api_key": "{{COHERE_API_KEY}}"
}
}
)$resp = $client->inference()->put([
"task_type" => "rerank",
"inference_id" => "my-rerank-model",
"body" => [
"service" => "cohere",
"service_settings" => [
"model_id" => "rerank-english-v3.0",
"api_key" => "{{COHERE_API_KEY}}",
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"service":"cohere","service_settings":{"model_id":"rerank-english-v3.0","api_key":"{{COHERE_API_KEY}}"}}' "$ELASTICSEARCH_URL/_inference/rerank/my-rerank-model"client.inference().put(p -> p
.inferenceId("my-rerank-model")
.taskType(TaskType.Rerank)
.inferenceConfig(i -> i
.service("cohere")
.serviceSettings(JsonData.fromJson("{\"model_id\":\"rerank-english-v3.0\",\"api_key\":\"{{COHERE_API_KEY}}\"}"))
)
);
{
"service": "cohere",
"service_settings": {
"model_id": "rerank-english-v3.0",
"api_key": "{{COHERE_API_KEY}}"
}
}The task type.
The only valid task type for the model to perform is completion.
Value is completion.
The unique identifier of the inference endpoint.
Specifies the amount of time to wait for the inference endpoint to be created.
Values are -1 or 0.
PUT _inference/completion/anthropic_completion
{
"service": "anthropic",
"service_settings": {
"api_key": "Anthropic-Api-Key",
"model_id": "Model-ID"
},
"task_settings": {
"max_tokens": 1024
}
}resp = client.inference.put(
task_type="completion",
inference_id="anthropic_completion",
inference_config={
"service": "anthropic",
"service_settings": {
"api_key": "Anthropic-Api-Key",
"model_id": "Model-ID"
},
"task_settings": {
"max_tokens": 1024
}
},
)const response = await client.inference.put({
task_type: "completion",
inference_id: "anthropic_completion",
inference_config: {
service: "anthropic",
service_settings: {
api_key: "Anthropic-Api-Key",
model_id: "Model-ID",
},
task_settings: {
max_tokens: 1024,
},
},
});response = client.inference.put(
task_type: "completion",
inference_id: "anthropic_completion",
body: {
"service": "anthropic",
"service_settings": {
"api_key": "Anthropic-Api-Key",
"model_id": "Model-ID"
},
"task_settings": {
"max_tokens": 1024
}
}
)$resp = $client->inference()->put([
"task_type" => "completion",
"inference_id" => "anthropic_completion",
"body" => [
"service" => "anthropic",
"service_settings" => [
"api_key" => "Anthropic-Api-Key",
"model_id" => "Model-ID",
],
"task_settings" => [
"max_tokens" => 1024,
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"service":"anthropic","service_settings":{"api_key":"Anthropic-Api-Key","model_id":"Model-ID"},"task_settings":{"max_tokens":1024}}' "$ELASTICSEARCH_URL/_inference/completion/anthropic_completion"client.inference().put(p -> p
.inferenceId("anthropic_completion")
.taskType(TaskType.Completion)
.inferenceConfig(i -> i
.service("anthropic")
.serviceSettings(JsonData.fromJson("{\"api_key\":\"Anthropic-Api-Key\",\"model_id\":\"Model-ID\"}"))
.taskSettings(JsonData.fromJson("{\"max_tokens\":1024}"))
)
);
{
"service": "anthropic",
"service_settings": {
"api_key": "Anthropic-Api-Key",
"model_id": "Model-ID"
},
"task_settings": {
"max_tokens": 1024
}
}The custom service gives more control over how to interact with external inference services that aren't explicitly supported through dedicated integrations.
The custom service gives you the ability to define the headers, url, query parameters, request body, and secrets.
The custom service supports the template replacement functionality, which enables you to define a template that can be replaced with the value associated with that key.
Templates are portions of a string that start with ${ and end with }.
The parameters secret_parameters and task_settings are checked for keys for template replacement. Template replacement is supported in the request, headers, url, and query_parameters.
If the definition (key) is not found for a template, an error message is returned.
In case of an endpoint definition like the following:
PUT _inference/text_embedding/test-text-embedding
{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<some api key>"
},
"url": "...endpoints.huggingface.cloud/v1/embeddings",
"headers": {
"Authorization": "Bearer ${api_key}",
"Content-Type": "application/json"
},
"request": "{\"input\": ${input}}",
"response": {
"json_parser": {
"text_embeddings":"$.data[*].embedding[*]"
}
}
}
}
To replace ${api_key} the secret_parameters and task_settings are checked for a key named api_key.
Templates should not be surrounded by quotes.
Pre-defined templates:
${input} refers to the array of input strings that comes from the input field of the subsequent inference requests.${input_type} refers to the input type translation values.${query} refers to the query field used specifically for reranking tasks.${top_n} refers to the top_n field available when performing rerank requests.${return_documents} refers to the return_documents field available when performing rerank requests.manage_inferenceThe type of the inference task that the model will perform.
Values are text_embedding, sparse_embedding, rerank, or completion.
The unique identifier of the inference endpoint.
PUT _inference/text_embedding/custom-embeddings
{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "https://api.openai.com/v1/embeddings",
"headers": {
"Authorization": "Bearer ${api_key}",
"Content-Type": "application/json;charset=utf-8"
},
"request": "{\"input\": ${input}, \"model\": \"text-embedding-3-small\"}",
"response": {
"json_parser": {
"text_embeddings": "$.data[*].embedding[*]"
}
}
}
}resp = client.inference.put(
task_type="text_embedding",
inference_id="custom-embeddings",
inference_config={
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "https://api.openai.com/v1/embeddings",
"headers": {
"Authorization": "Bearer ${api_key}",
"Content-Type": "application/json;charset=utf-8"
},
"request": "{\"input\": ${input}, \"model\": \"text-embedding-3-small\"}",
"response": {
"json_parser": {
"text_embeddings": "$.data[*].embedding[*]"
}
}
}
},
)const response = await client.inference.put({
task_type: "text_embedding",
inference_id: "custom-embeddings",
inference_config: {
service: "custom",
service_settings: {
secret_parameters: {
api_key: "<api key>",
},
url: "https://api.openai.com/v1/embeddings",
headers: {
Authorization: "Bearer ${api_key}",
"Content-Type": "application/json;charset=utf-8",
},
request: '{"input": ${input}, "model": "text-embedding-3-small"}',
response: {
json_parser: {
text_embeddings: "$.data[*].embedding[*]",
},
},
},
},
});response = client.inference.put(
task_type: "text_embedding",
inference_id: "custom-embeddings",
body: {
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "https://api.openai.com/v1/embeddings",
"headers": {
"Authorization": "Bearer ${api_key}",
"Content-Type": "application/json;charset=utf-8"
},
"request": "{\"input\": ${input}, \"model\": \"text-embedding-3-small\"}",
"response": {
"json_parser": {
"text_embeddings": "$.data[*].embedding[*]"
}
}
}
}
)$resp = $client->inference()->put([
"task_type" => "text_embedding",
"inference_id" => "custom-embeddings",
"body" => [
"service" => "custom",
"service_settings" => [
"secret_parameters" => [
"api_key" => "<api key>",
],
"url" => "https://api.openai.com/v1/embeddings",
"headers" => [
"Authorization" => "Bearer ${api_key}",
"Content-Type" => "application/json;charset=utf-8",
],
"request" => "{\"input\": ${input}, \"model\": \"text-embedding-3-small\"}",
"response" => [
"json_parser" => [
"text_embeddings" => "$.data[*].embedding[*]",
],
],
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"service":"custom","service_settings":{"secret_parameters":{"api_key":"<api key>"},"url":"https://api.openai.com/v1/embeddings","headers":{"Authorization":"Bearer ${api_key}","Content-Type":"application/json;charset=utf-8"},"request":"{\"input\": ${input}, \"model\": \"text-embedding-3-small\"}","response":{"json_parser":{"text_embeddings":"$.data[*].embedding[*]"}}}}' "$ELASTICSEARCH_URL/_inference/text_embedding/custom-embeddings"{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "https://api.openai.com/v1/embeddings",
"headers": {
"Authorization": "Bearer ${api_key}",
"Content-Type": "application/json;charset=utf-8"
},
"request": "{\"input\": ${input}, \"model\": \"text-embedding-3-small\"}",
"response": {
"json_parser": {
"text_embeddings": "$.data[*].embedding[*]"
}
}
}
}{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "https://api.cohere.com/v2/rerank",
"headers": {
"Authorization": "bearer ${api_key}",
"Content-Type": "application/json"
},
"request": "{\"documents\": ${input}, \"query\": ${query}, \"model\": \"rerank-v3.5\"}",
"response": {
"json_parser": {
"reranked_index":"$.results[*].index",
"relevance_score":"$.results[*].relevance_score"
}
}
}
}{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "https://api.cohere.com/v2/embed",
"headers": {
"Authorization": "bearer ${api_key}",
"Content-Type": "application/json"
},
"request": "{\"texts\": ${input}, \"model\": \"embed-v4.0\", \"input_type\": ${input_type}}",
"response": {
"json_parser": {
"text_embeddings":"$.embeddings.float[*]"
}
},
"input_type": {
"translation": {
"ingest": "search_document",
"search": "search_query"
},
"default": "search_document"
}
}
}{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "https://api.jina.ai/v1/rerank",
"headers": {
"Content-Type": "application/json",
"Authorization": "Bearer ${api_key}"
},
"request": "{\"model\": \"jina-reranker-v2-base-multilingual\",\"query\": ${query},\"documents\":${input}}",
"response": {
"json_parser": {
"relevance_score": "$.results[*].relevance_score",
"reranked_index": "$.results[*].index"
}
}
}
}{
"service": "custom",
"service_settings": {
"secret_parameters": {
"api_key": "<api key>"
},
"url": "<dedicated inference endpoint on HF>/v1/embeddings",
"headers": {
"Authorization": "Bearer ${api_key}",
"Content-Type": "application/json"
},
"request": "{\"input\": ${input}}",
"response": {
"json_parser": {
"text_embeddings":"$.data[*].embedding[*]"
}
}
}
}The type of the inference task that the model will perform.
Values are completion or text_embedding.
The unique identifier of the inference endpoint.
Specifies the amount of time to wait for the inference endpoint to be created.
Values are -1 or 0.
PUT _inference/completion/google_ai_studio_completion
{
"service": "googleaistudio",
"service_settings": {
"api_key": "api-key",
"model_id": "model-id"
}
}resp = client.inference.put(
task_type="completion",
inference_id="google_ai_studio_completion",
inference_config={
"service": "googleaistudio",
"service_settings": {
"api_key": "api-key",
"model_id": "model-id"
}
},
)const response = await client.inference.put({
task_type: "completion",
inference_id: "google_ai_studio_completion",
inference_config: {
service: "googleaistudio",
service_settings: {
api_key: "api-key",
model_id: "model-id",
},
},
});response = client.inference.put(
task_type: "completion",
inference_id: "google_ai_studio_completion",
body: {
"service": "googleaistudio",
"service_settings": {
"api_key": "api-key",
"model_id": "model-id"
}
}
)$resp = $client->inference()->put([
"task_type" => "completion",
"inference_id" => "google_ai_studio_completion",
"body" => [
"service" => "googleaistudio",
"service_settings" => [
"api_key" => "api-key",
"model_id" => "model-id",
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"service":"googleaistudio","service_settings":{"api_key":"api-key","model_id":"model-id"}}' "$ELASTICSEARCH_URL/_inference/completion/google_ai_studio_completion"client.inference().put(p -> p
.inferenceId("google_ai_studio_completion")
.taskType(TaskType.Completion)
.inferenceConfig(i -> i
.service("googleaistudio")
.serviceSettings(JsonData.fromJson("{\"api_key\":\"api-key\",\"model_id\":\"model-id\"}"))
)
);
{
"service": "googleaistudio",
"service_settings": {
"api_key": "api-key",
"model_id": "model-id"
}
}Get real-time responses for completion tasks by delivering answers incrementally, reducing response times during computation. This API works only with the completion task type.
IMPORTANT: The inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.
This API requires the monitor_inference cluster privilege (the built-in inference_admin and inference_user roles grant this privilege). You must use a client that supports streaming.
monitor_inferenceThe amount of time to wait for the inference request to complete.
Values are -1 or 0.
POST _inference/completion/openai-completion/_stream
{
"input": "What is Elastic?"
}resp = client.inference.stream_completion(
inference_id="openai-completion",
input="What is Elastic?",
)const response = await client.inference.streamCompletion({
inference_id: "openai-completion",
input: "What is Elastic?",
});response = client.inference.stream_completion(
inference_id: "openai-completion",
body: {
"input": "What is Elastic?"
}
)$resp = $client->inference()->streamCompletion([
"inference_id" => "openai-completion",
"body" => [
"input" => "What is Elastic?",
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"input":"What is Elastic?"}' "$ELASTICSEARCH_URL/_inference/completion/openai-completion/_stream"client.inference().streamCompletion(s -> s
.inferenceId("openai-completion")
.input("What is Elastic?")
);
{
"input": "What is Elastic?"
}All methods and paths for this operation:
Modify task_settings, secrets (within service_settings), or num_allocations for an inference endpoint, depending on the specific endpoint service and task_type.
IMPORTANT: The inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.
manage_inferenceThe type of inference task that the model performs.
Values are sparse_embedding, text_embedding, rerank, completion, or chat_completion.
The unique identifier of the inference endpoint.
PUT _inference/my-inference-endpoint/_update
{
"service_settings": {
"api_key": "<API_KEY>"
}
}resp = client.inference.update(
inference_id="my-inference-endpoint",
inference_config={
"service_settings": {
"api_key": "<API_KEY>"
}
},
)const response = await client.inference.update({
inference_id: "my-inference-endpoint",
inference_config: {
service_settings: {
api_key: "<API_KEY>",
},
},
});response = client.inference.update(
inference_id: "my-inference-endpoint",
body: {
"service_settings": {
"api_key": "<API_KEY>"
}
}
)$resp = $client->inference()->update([
"inference_id" => "my-inference-endpoint",
"body" => [
"service_settings" => [
"api_key" => "<API_KEY>",
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"service_settings":{"api_key":"<API_KEY>"}}' "$ELASTICSEARCH_URL/_inference/my-inference-endpoint/_update"{
"service_settings": {
"api_key": "<API_KEY>"
}
}