Get an autoscaling policy
Generally available; Added in 7.11.0
NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.
GET /_autoscaling/policy/my_autoscaling_policy
resp = client.autoscaling.get_autoscaling_policy(
name="my_autoscaling_policy",
)const response = await client.autoscaling.getAutoscalingPolicy({
name: "my_autoscaling_policy",
});response = client.autoscaling.get_autoscaling_policy(
name: "my_autoscaling_policy"
)$resp = $client->autoscaling()->getAutoscalingPolicy([
"name" => "my_autoscaling_policy",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_autoscaling/policy/my_autoscaling_policy"client.autoscaling().getAutoscalingPolicy(g -> g
.name("my_autoscaling_policy")
);
{
"roles": <roles>,
"deciders": <deciders>
}
Create or update an autoscaling policy
Generally available; Added in 7.11.0
NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0.
Body
Required
-
Decider settings.
External documentation
PUT /_autoscaling/policy/<name>
{
"roles": [],
"deciders": {
"fixed": {
}
}
}resp = client.autoscaling.put_autoscaling_policy(
name="<name>",
policy={
"roles": [],
"deciders": {
"fixed": {}
}
},
)const response = await client.autoscaling.putAutoscalingPolicy({
name: "<name>",
policy: {
roles: [],
deciders: {
fixed: {},
},
},
});response = client.autoscaling.put_autoscaling_policy(
name: "<name>",
body: {
"roles": [],
"deciders": {
"fixed": {}
}
}
)$resp = $client->autoscaling()->putAutoscalingPolicy([
"name" => "<name>",
"body" => [
"roles" => array(
),
"deciders" => [
"fixed" => new ArrayObject([]),
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"roles":[],"deciders":{"fixed":{}}}' "$ELASTICSEARCH_URL/_autoscaling/policy/<name>"client.autoscaling().putAutoscalingPolicy(p -> p
.name("<name>")
.policy(po -> po
.deciders("fixed", JsonData.fromJson("{}"))
)
);
{
"roles": [],
"deciders": {
"fixed": {
}
}
}{
"roles" : [ "data_hot" ],
"deciders": {
"fixed": {
}
}
}{
"acknowledged": true
}
Update voting configuration exclusions
Generally available; Added in 7.0.0
Update the cluster voting config exclusions by node IDs or node names. By default, if there are more than three master-eligible nodes in the cluster and you remove fewer than half of the master-eligible nodes in the cluster at once, the voting configuration automatically shrinks. If you want to shrink the voting configuration to contain fewer than three nodes or to remove half or more of the master-eligible nodes in the cluster at once, use this API to remove departing nodes from the voting configuration manually. The API adds an entry for each specified node to the cluster’s voting configuration exclusions list. It then waits until the cluster has reconfigured its voting configuration to exclude the specified nodes.
Clusters should have no voting configuration exclusions in normal operation.
Once the excluded nodes have stopped, clear the voting configuration exclusions with DELETE /_cluster/voting_config_exclusions.
This API waits for the nodes to be fully removed from the cluster before it returns.
If your cluster has voting configuration exclusions for nodes that you no longer intend to remove, use DELETE /_cluster/voting_config_exclusions?wait_for_removal=false to clear the voting configuration exclusions without waiting for the nodes to leave the cluster.
A response to POST /_cluster/voting_config_exclusions with an HTTP status code of 200 OK guarantees that the node has been removed from the voting configuration and will not be reinstated until the voting configuration exclusions are cleared by calling DELETE /_cluster/voting_config_exclusions.
If the call to POST /_cluster/voting_config_exclusions fails or returns a response with an HTTP status code other than 200 OK then the node may not have been removed from the voting configuration.
In that case, you may safely retry the call.
NOTE: Voting exclusions are required only when you remove at least half of the master-eligible nodes from a cluster in a short time period. They are not required when removing master-ineligible nodes or when removing fewer than half of the master-eligible nodes.
Query parameters
-
A comma-separated list of the names of the nodes to exclude from the voting configuration. If specified, you may not also specify node_ids.
-
A comma-separated list of the persistent ids of the nodes to exclude from the voting configuration. If specified, you may not also specify node_names.
-
Period to wait for a connection to the master node.
Values are
-1or0. -
When adding a voting configuration exclusion, the API waits for the specified nodes to be excluded from the voting configuration before returning. If the timeout expires before the appropriate condition is satisfied, the request fails and returns an error.
Values are
-1or0.
curl \
--request POST 'http://api.example.com/_cluster/voting_config_exclusions' \
--header "Authorization: $API_KEY"GET _connector/my-connector-id
resp = client.connector.get(
connector_id="my-connector-id",
)const response = await client.connector.get({
connector_id: "my-connector-id",
});response = client.connector.get(
connector_id: "my-connector-id"
)$resp = $client->connector()->get([
"connector_id" => "my-connector-id",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/my-connector-id"client.connector().get(g -> g
.connectorId("my-connector-id")
);
Path parameters
-
Name of the data stream, which must meet the following criteria: Lowercase only; Cannot include
\,/,*,?,",<,>,|,,,#,:, or a space character; Cannot start with-,_,+, or.ds-; Cannot be.or..; Cannot be longer than 255 bytes. Multi-byte characters count towards this limit faster.
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0.
PUT _data_stream/logs-foo-bar
resp = client.indices.create_data_stream(
name="logs-foo-bar",
)const response = await client.indices.createDataStream({
name: "logs-foo-bar",
});response = client.indices.create_data_stream(
name: "logs-foo-bar"
)$resp = $client->indices()->createDataStream([
"name" => "logs-foo-bar",
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/logs-foo-bar"client.indices().createDataStream(c -> c
.name("logs-foo-bar")
);
Path parameters
-
Comma-separated list of data streams to delete. Wildcard (
*) expressions are supported.
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0. -
Type of data stream that wildcard patterns can match. Supports comma-separated values,such as
open,hidden.Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined withopen,closed, orboth.none: Wildcard expressions are not accepted.
Values are
all,open,closed,hidden, ornone.
DELETE _data_stream/my-data-stream
resp = client.indices.delete_data_stream(
name="my-data-stream",
)const response = await client.indices.deleteDataStream({
name: "my-data-stream",
});response = client.indices.delete_data_stream(
name: "my-data-stream"
)$resp = $client->indices()->deleteDataStream([
"name" => "my-data-stream",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/my-data-stream"client.indices().deleteDataStream(d -> d
.name("my-data-stream")
);
Update data streams
Generally available; Added in 7.16.0
Performs one or more data stream modification actions in a single atomic operation.
POST _data_stream/_modify
{
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}resp = client.indices.modify_data_stream(
actions=[
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
],
)const response = await client.indices.modifyDataStream({
actions: [
{
remove_backing_index: {
data_stream: "my-data-stream",
index: ".ds-my-data-stream-2023.07.26-000001",
},
},
{
add_backing_index: {
data_stream: "my-data-stream",
index: ".ds-my-data-stream-2023.07.26-000001-downsample",
},
},
],
});response = client.indices.modify_data_stream(
body: {
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}
)$resp = $client->indices()->modifyDataStream([
"body" => [
"actions" => array(
[
"remove_backing_index" => [
"data_stream" => "my-data-stream",
"index" => ".ds-my-data-stream-2023.07.26-000001",
],
],
[
"add_backing_index" => [
"data_stream" => "my-data-stream",
"index" => ".ds-my-data-stream-2023.07.26-000001-downsample",
],
],
),
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"actions":[{"remove_backing_index":{"data_stream":"my-data-stream","index":".ds-my-data-stream-2023.07.26-000001"}},{"add_backing_index":{"data_stream":"my-data-stream","index":".ds-my-data-stream-2023.07.26-000001-downsample"}}]}' "$ELASTICSEARCH_URL/_data_stream/_modify"client.indices().modifyDataStream(m -> m
.actions(List.of(Action.of(a -> a
.removeBackingIndex(r -> r
.dataStream("my-data-stream")
.index(".ds-my-data-stream-2023.07.26-000001")
)),Action.of(ac -> ac
.addBackingIndex(ad -> ad
.dataStream("my-data-stream")
.index(".ds-my-data-stream-2023.07.26-000001-downsample")
))))
);
{
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}
Delete a document
Generally available
Remove a JSON document from the specified index.
NOTE: You cannot send deletion requests directly to a data stream. To delete a document in a data stream, you must target the backing index containing the document.
Optimistic concurrency control
Delete operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters.
If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409.
Versioning
Each document indexed is versioned.
When deleting a document, the version can be specified to make sure the relevant document you are trying to delete is actually being deleted and it has not changed in the meantime.
Every write operation run on a document, deletes included, causes its version to be incremented.
The version number of a deleted document remains available for a short time after deletion to allow for control of concurrent operations.
The length of time for which a deleted document's version remains available is determined by the index.gc_deletes index setting.
Routing
If routing is used during indexing, the routing value also needs to be specified to delete a document.
If the _routing mapping is set to required and no routing value is specified, the delete API throws a RoutingMissingException and rejects the request.
For example:
DELETE /my-index-000001/_doc/1?routing=shard-1
This request deletes the document with ID 1, but it is routed based on the user. The document is not deleted if the correct routing is not specified.
Distributed
The delete operation gets hashed into a specific shard ID. It then gets redirected into the primary shard within that ID group and replicated (if needed) to shard replicas within that ID group.
Required authorization
- Index privileges:
delete
Query parameters
-
Only perform the operation if the document has this primary term.
-
Only perform the operation if the document has this sequence number.
-
If
true, Elasticsearch refreshes the affected shards to make this operation visible to search. Ifwait_for, it waits for a refresh to make this operation visible to search. Iffalse, it does nothing with refreshes.Values are
true,false, orwait_for. -
A custom value used to route operations to a specific shard.
-
The period to wait for active shards.
This parameter is useful for situations where the primary shard assigned to perform the delete operation might not be available when the delete operation runs. Some reasons for this might be that the primary shard is currently recovering from a store or undergoing relocation. By default, the delete operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error.
Values are
-1or0. -
An explicit version number for concurrency control. It must match the current version of the document for the request to succeed.
-
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document. NOTE: Theexternal_gteversion type is meant for special use cases and should be used with care. If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.
Values are
internal,external,external_gte, orforce. -
The minimum number of shard copies that must be active before proceeding with the operation. You can set it to
allor any positive integer up to the total number of shards in the index (number_of_replicas+1). The default value of1means it waits for each primary shard to be active.Values are
allorindex-setting.
DELETE /my-index-000001/_doc/1
resp = client.delete(
index="my-index-000001",
id="1",
)const response = await client.delete({
index: "my-index-000001",
id: 1,
});response = client.delete(
index: "my-index-000001",
id: "1"
)$resp = $client->delete([
"index" => "my-index-000001",
"id" => "1",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_doc/1"client.delete(d -> d
.id("1")
.index("my-index-000001")
);
{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
},
"_index": "my-index-000001",
"_id": "1",
"_version": 2,
"_primary_term": 1,
"_seq_no": 5,
"result": "deleted"
}
Delete documents
Generally available; Added in 5.0.0
Deletes documents that match the specified query.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or alias:
readdeleteorwrite
You can specify the query criteria in the request URI or the request body using the same syntax as the search API. When you submit a delete by query request, Elasticsearch gets a snapshot of the data stream or index when it begins processing the request and deletes matching documents using internal versioning. If a document changes between the time that the snapshot is taken and the delete operation is processed, it results in a version conflict and the delete operation fails.
NOTE: Documents with a version equal to 0 cannot be deleted using delete by query because internal versioning does not support 0 as a valid version number.
While processing a delete by query request, Elasticsearch performs multiple search requests sequentially to find all of the matching documents to delete. A bulk delete request is performed for each batch of matching documents. If a search or bulk request is rejected, the requests are retried up to 10 times, with exponential back off. If the maximum retry limit is reached, processing halts and all failed requests are returned in the response. Any delete requests that completed successfully still stick, they are not rolled back.
You can opt to count version conflicts instead of halting and returning by setting conflicts to proceed.
Note that if you opt to count version conflicts the operation could attempt to delete more documents from the source than max_docs until it has successfully deleted max_docs documents, or it has gone through every document in the source query.
Throttling delete requests
To control the rate at which delete by query issues batches of delete operations, you can set requests_per_second to any positive decimal number.
This pads each batch with a wait time to throttle the rate.
Set requests_per_second to -1 to disable throttling.
Throttling uses a wait time between batches so that the internal scroll requests can be given a timeout that takes the request padding into account.
The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing.
By default the batch size is 1000, so if requests_per_second is set to 500:
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
Since the batch is issued as a single _bulk request, large batch sizes cause Elasticsearch to create many requests and wait before starting the next set.
This is "bursty" instead of "smooth".
Slicing
Delete by query supports sliced scroll to parallelize the delete process. This can improve efficiency and provide a convenient way to break the request down into smaller parts.
Setting slices to auto lets Elasticsearch choose the number of slices to use.
This setting will use one slice per shard, up to a certain limit.
If there are multiple source data streams or indices, it will choose the number of slices based on the index or backing index with the smallest number of shards.
Adding slices to the delete by query operation creates sub-requests which means it has some quirks:
- You can see these requests in the tasks APIs. These sub-requests are "child" tasks of the task for the request with slices.
- Fetching the status of the task for the request with slices only contains the status of completed slices.
- These sub-requests are individually addressable for things like cancellation and rethrottling.
- Rethrottling the request with
sliceswill rethrottle the unfinished sub-request proportionally. - Canceling the request with
sliceswill cancel each sub-request. - Due to the nature of
sliceseach sub-request won't get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution. - Parameters like
requests_per_secondandmax_docson a request withslicesare distributed proportionally to each sub-request. Combine that with the earlier point about distribution being uneven and you should conclude that usingmax_docswithslicesmight not result in exactlymax_docsdocuments being deleted. - Each sub-request gets a slightly different snapshot of the source data stream or index though these are all taken at approximately the same time.
If you're slicing manually or otherwise tuning automatic slicing, keep in mind that:
- Query performance is most efficient when the number of slices is equal to the number of shards in the index or backing index. If that number is large (for example, 500), choose a lower number as too many
sliceshurts performance. Settingsliceshigher than the number of shards generally does not improve efficiency and adds overhead. - Delete performance scales linearly across available resources with the number of slices.
Whether query or delete performance dominates the runtime depends on the documents being reindexed and cluster resources.
Cancel a delete by query operation
Any delete by query can be canceled using the task cancel API. For example:
POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel
The task ID can be found by using the get tasks API.
Cancellation should happen quickly but might take a few seconds. The get task status API will continue to list the delete by query task until this task checks that it has been cancelled and terminates itself.
Required authorization
- Index privileges:
read,delete
Path parameters
-
A comma-separated list of data streams, indices, and aliases to search. It supports wildcards (
*). To search all data streams or indices, omit this parameter or use*or_all.
Query parameters
-
If
false, the request returns an error if any wildcard expression, index alias, or_allvalue targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*returns an error if an index starts withfoobut no index starts withbar. -
Analyzer to use for the query string. This parameter can be used only when the
qquery string parameter is specified. -
If
true, wildcard and prefix queries are analyzed. This parameter can be used only when theqquery string parameter is specified. -
What to do if delete by query hits version conflicts:
abortorproceed.Supported values include:
abort: Stop reindexing if there are conflicts.proceed: Continue reindexing even if there are conflicts.
Values are
abortorproceed. -
The default operator for query string query:
ANDorOR. This parameter can be used only when theqquery string parameter is specified.Values are
and,AND,or, orOR. -
The field to use as default where no field prefix is given in the query string. This parameter can be used only when the
qquery string parameter is specified. -
The type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values, such as
open,hidden.Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined withopen,closed, orboth.none: Wildcard expressions are not accepted.
Values are
all,open,closed,hidden, ornone. -
Skips the specified number of documents.
-
If
true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored. This parameter can be used only when theqquery string parameter is specified. -
The maximum number of documents to process. Defaults to all documents. When set to a value less then or equal to
scroll_size, a scroll will not be used to retrieve the results for the operation. -
The node or shard the operation should be performed on. It is random by default.
-
If
true, Elasticsearch refreshes all shards involved in the delete by query after the request completes. This is different than the delete API'srefreshparameter, which causes just the shard that received the delete request to be refreshed. Unlike the delete API, it does not supportwait_for. -
If
true, the request cache is used for this request. Defaults to the index-level setting. -
The throttle for this request in sub-requests per second.
-
A custom value used to route operations to a specific shard.
-
A query in the Lucene query string syntax.
-
The period to retain the search context for scrolling.
Values are
-1or0. -
The size of the scroll request that powers the operation.
-
The explicit timeout for each search request. It defaults to no timeout.
Values are
-1or0. -
The type of the search operation. Available options include
query_then_fetchanddfs_query_then_fetch.Supported values include:
query_then_fetch: Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.dfs_query_then_fetch: Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.
Values are
query_then_fetchordfs_query_then_fetch. -
The number of slices this task should be divided into.
Value is
auto. -
A comma-separated list of
<field>:<direction>pairs. -
The specific
tagof the request for logging and statistical purposes. -
The maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.
Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.
-
The period each deletion request waits for active shards.
Values are
-1or0. -
If
true, returns the document version as part of a hit. -
The number of shard copies that must be active before proceeding with the operation. Set to
allor any positive integer up to the total number of shards in the index (number_of_replicas+1). Thetimeoutvalue controls how long each write request waits for unavailable shards to become available.Values are
allorindex-setting. -
If
true, the request blocks until the operation is complete. Iffalse, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task. Elasticsearch creates a record of this task as a document at.tasks/task/${taskId}. When you are done with a task, you should delete the task document so Elasticsearch can reclaim the space.
Body
Required
-
The maximum number of documents to delete.
-
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
External documentation
POST /my-index-000001,my-index-000002/_delete_by_query
{
"query": {
"match_all": {}
}
}resp = client.delete_by_query(
index="my-index-000001,my-index-000002",
query={
"match_all": {}
},
)const response = await client.deleteByQuery({
index: "my-index-000001,my-index-000002",
query: {
match_all: {},
},
});response = client.delete_by_query(
index: "my-index-000001,my-index-000002",
body: {
"query": {
"match_all": {}
}
}
)$resp = $client->deleteByQuery([
"index" => "my-index-000001,my-index-000002",
"body" => [
"query" => [
"match_all" => new ArrayObject([]),
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"query":{"match_all":{}}}' "$ELASTICSEARCH_URL/my-index-000001,my-index-000002/_delete_by_query"client.deleteByQuery(d -> d
.index(List.of("my-index-000001","my-index-000002"))
.query(q -> q
.matchAll(m -> m)
)
);
{
"query": {
"match_all": {}
}
}{
"query": {
"term": {
"user.id": "kimchy"
}
},
"max_docs": 1
}{
"slice": {
"id": 0,
"max": 2
},
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}{
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}{
"took" : 147,
"timed_out": false,
"total": 119,
"deleted": 119,
"batches": 1,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"failures" : [ ]
}
Check for a document source
Generally available; Added in 5.4.0
Check whether a document source exists in an index. For example:
HEAD my-index-000001/_source/1
A document's source is not available if it is disabled in the mapping.
Required authorization
- Index privileges:
read
Path parameters
-
A comma-separated list of data streams, indices, and aliases. It supports wildcards (
*). -
A unique identifier for the document.
Query parameters
-
The node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.
-
If
true, the request is real-time as opposed to near-real-time. -
If
true, the request refreshes the relevant shards before retrieving the document. Setting it totrueshould be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing). -
A custom value used to route operations to a specific shard.
-
Indicates whether to return the
_sourcefield (trueorfalse) or lists the fields to return. -
A comma-separated list of source fields to exclude in the response.
-
A comma-separated list of source fields to include in the response.
-
The version number for concurrency control. It must match the current version of the document for the request to succeed.
-
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document. NOTE: Theexternal_gteversion type is meant for special use cases and should be used with care. If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.
Values are
internal,external,external_gte, orforce.
HEAD my-index-000001/_source/1
resp = client.exists_source(
index="my-index-000001",
id="1",
)const response = await client.existsSource({
index: "my-index-000001",
id: 1,
});response = client.exists_source(
index: "my-index-000001",
id: "1"
)$resp = $client->existsSource([
"index" => "my-index-000001",
"id" => "1",
]);curl --head -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_source/1"client.existsSource(e -> e
.id("1")
.index("my-index-000001")
);
Throttle a reindex operation
Generally available; Added in 2.4.0
Change the number of requests per second for a particular reindex operation. For example:
POST _reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
Rethrottling that speeds up the query takes effect immediately. Rethrottling that slows down the query will take effect after completing the current batch. This behavior prevents scroll timeouts.
POST _reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1
resp = client.reindex_rethrottle(
task_id="r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second="-1",
)const response = await client.reindexRethrottle({
task_id: "r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second: "-1",
});response = client.reindex_rethrottle(
task_id: "r1A2WoRbTwKZ516z6NEs5A:36619",
requests_per_second: "-1"
)$resp = $client->reindexRethrottle([
"task_id" => "r1A2WoRbTwKZ516z6NEs5A:36619",
"requests_per_second" => "-1",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_reindex/r1A2WoRbTwKZ516z6NEs5A:36619/_rethrottle?requests_per_second=-1"client.reindexRethrottle(r -> r
.requestsPerSecond(-1.0F)
.taskId("r1A2WoRbTwKZ516z6NEs5A:36619")
);
Explore graph analytics
Generally available
All methods and paths for this operation:
Extract and summarize information about the documents and terms in an Elasticsearch data stream or index.
The easiest way to understand the behavior of this API is to use the Graph UI to explore connections.
An initial request to the _explore API contains a seed query that identifies the documents of interest and specifies the fields that define the vertices and connections you want to include in the graph.
Subsequent requests enable you to spider out from one more vertices of interest.
You can exclude vertices that have already been returned.
Query parameters
-
Custom value used to route operations to a specific shard.
-
Specifies the period of time to wait for a response from each shard. If no response is received before the timeout expires, the request fails and returns an error. Defaults to no timeout.
Values are
-1or0.
Body
-
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
External documentation -
Specifies one or more fields that contain the terms you want to include in the graph as vertices.
POST clicklogs/_graph/explore
{
"query": {
"match": {
"query.raw": "midi"
}
},
"vertices": [
{
"field": "product"
}
],
"connections": {
"vertices": [
{
"field": "query.raw"
}
]
}
}resp = client.graph.explore(
index="clicklogs",
query={
"match": {
"query.raw": "midi"
}
},
vertices=[
{
"field": "product"
}
],
connections={
"vertices": [
{
"field": "query.raw"
}
]
},
)const response = await client.graph.explore({
index: "clicklogs",
query: {
match: {
"query.raw": "midi",
},
},
vertices: [
{
field: "product",
},
],
connections: {
vertices: [
{
field: "query.raw",
},
],
},
});response = client.graph.explore(
index: "clicklogs",
body: {
"query": {
"match": {
"query.raw": "midi"
}
},
"vertices": [
{
"field": "product"
}
],
"connections": {
"vertices": [
{
"field": "query.raw"
}
]
}
}
)$resp = $client->graph()->explore([
"index" => "clicklogs",
"body" => [
"query" => [
"match" => [
"query.raw" => "midi",
],
],
"vertices" => array(
[
"field" => "product",
],
),
"connections" => [
"vertices" => array(
[
"field" => "query.raw",
],
),
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"query":{"match":{"query.raw":"midi"}},"vertices":[{"field":"product"}],"connections":{"vertices":[{"field":"query.raw"}]}}' "$ELASTICSEARCH_URL/clicklogs/_graph/explore"client.graph().explore(e -> e
.connections(c -> c
.vertices(v -> v
.field("query.raw")
)
)
.index("clicklogs")
.query(q -> q
.match(m -> m
.field("query.raw")
.query(FieldValue.of("midi"))
)
)
.vertices(v -> v
.field("product")
)
);
{
"query": {
"match": {
"query.raw": "midi"
}
},
"vertices": [
{
"field": "product"
}
],
"connections": {
"vertices": [
{
"field": "query.raw"
}
]
}
}
Create an index
Generally available
You can use the create index API to add a new index to an Elasticsearch cluster. When creating an index, you can specify the following:
- Settings for the index.
- Mappings for fields in the index.
- Index aliases
Wait for active shards
By default, index creation will only return a response to the client when the primary copies of each shard have been started, or the request times out.
The index creation response will indicate what happened.
For example, acknowledged indicates whether the index was successfully created in the cluster, while shards_acknowledged indicates whether the requisite number of shard copies were started for each shard in the index before timing out.
Note that it is still possible for either acknowledged or shards_acknowledged to be false, but for the index creation to be successful.
These values simply indicate whether the operation completed before the timeout.
If acknowledged is false, the request timed out before the cluster state was updated with the newly created index, but it probably will be created sometime soon.
If shards_acknowledged is false, then the request timed out before the requisite number of shards were started (by default just the primaries), even if the cluster state was successfully updated to reflect the newly created index (that is to say, acknowledged is true).
You can change the default of only waiting for the primary shards to start through the index setting index.write.wait_for_active_shards.
Note that changing this setting will also affect the wait_for_active_shards value on all subsequent write operations.
Required authorization
- Index privileges:
create_index,manage
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0. -
The number of shard copies that must be active before proceeding with the operation. Set to
allor any positive integer up to the total number of shards in the index (number_of_replicas+1).Values are
allorindex-setting.
Body
-
Aliases for the index.
-
Index settings
PUT /my-index-000001
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}resp = client.indices.create(
index="my-index-000001",
settings={
"number_of_shards": 3,
"number_of_replicas": 2
},
)const response = await client.indices.create({
index: "my-index-000001",
settings: {
number_of_shards: 3,
number_of_replicas: 2,
},
});response = client.indices.create(
index: "my-index-000001",
body: {
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}
)$resp = $client->indices()->create([
"index" => "my-index-000001",
"body" => [
"settings" => [
"number_of_shards" => 3,
"number_of_replicas" => 2,
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"settings":{"number_of_shards":3,"number_of_replicas":2}}' "$ELASTICSEARCH_URL/my-index-000001"client.indices().create(c -> c
.index("my-index-000001")
.settings(s -> s
.numberOfShards("3")
.numberOfReplicas("2")
)
);
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 2
}
}{
"settings": {
"number_of_shards": 1
},
"mappings": {
"properties": {
"field1": { "type": "text" }
}
}
}{
"aliases": {
"alias_1": {},
"alias_2": {
"filter": {
"term": {
"user.id": "kimchy"
}
},
"routing": "shard-1"
}
}
}
Check indices
Generally available
Check if one or more indices, index aliases, or data streams exist.
Query parameters
-
If
false, the request returns an error if any wildcard expression, index alias, or_allvalue targets only missing or closed indices. This behavior applies even if the request targets other open indices. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden.Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined withopen,closed, orboth.none: Wildcard expressions are not accepted.
Values are
all,open,closed,hidden, ornone. -
If
true, returns settings in flat format. -
If
true, return all default settings in the response. -
If
true, the request retrieves information from the local node only.
HEAD my-data-stream
resp = client.indices.exists(
index="my-data-stream",
)const response = await client.indices.exists({
index: "my-data-stream",
});response = client.indices.exists(
index: "my-data-stream"
)$resp = $client->indices()->exists([
"index" => "my-data-stream",
]);curl --head -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-data-stream"client.indices().exists(e -> e
.index("my-data-stream")
);
Path parameters
-
Comma-separated list of data streams or indices used to limit the request. Supports wildcards (
*). -
Comma-separated list of aliases to remove. Supports wildcards (
*). To remove all aliases, use*or_all.
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0.
DELETE my-data-stream/_alias/my-alias
resp = client.indices.delete_alias(
index="my-data-stream",
name="my-alias",
)const response = await client.indices.deleteAlias({
index: "my-data-stream",
name: "my-alias",
});response = client.indices.delete_alias(
index: "my-data-stream",
name: "my-alias"
)$resp = $client->indices()->deleteAlias([
"index" => "my-data-stream",
"name" => "my-alias",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-data-stream/_alias/my-alias"client.indices().deleteAlias(d -> d
.index("my-data-stream")
.name("my-alias")
);
Flush data streams or indices
Generally available
All methods and paths for this operation:
Flushing a data stream or index is the process of making sure that any data that is currently only stored in the transaction log is also permanently stored in the Lucene index. When restarting, Elasticsearch replays any unflushed operations from the transaction log into the Lucene index to bring it back into the state that it was in before the restart. Elasticsearch automatically triggers flushes as needed, using heuristics that trade off the size of the unflushed transaction log against the cost of performing each flush.
After each operation has been flushed it is permanently stored in the Lucene index. This may mean that there is no need to maintain an additional copy of it in the transaction log. The transaction log is made up of multiple files, called generations, and Elasticsearch will delete any generation files when they are no longer needed, freeing up disk space.
It is also possible to trigger a flush on one or more indices using the flush API, although it is rare for users to need to call this API directly. If you call the flush API after indexing some documents then a successful response indicates that Elasticsearch has flushed all the documents that were indexed before the flush API was called.
Required authorization
- Index privileges:
maintenance
Path parameters
-
Comma-separated list of data streams, indices, and aliases to flush. Supports wildcards (
*). To flush all data streams and indices, omit this parameter or use*or_all.
Query parameters
-
If
false, the request returns an error if any wildcard expression, index alias, or_allvalue targets only missing or closed indices. This behavior applies even if the request targets other open indices. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such as
open,hidden.Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined withopen,closed, orboth.none: Wildcard expressions are not accepted.
Values are
all,open,closed,hidden, ornone. -
If
true, the request forces a flush even if there are no changes to commit to the index. -
If
true, the flush operation blocks until execution when another flush operation is running. Iffalse, Elasticsearch returns an error if you request a flush when another flush operation is running.
POST /_flush
resp = client.indices.flush()const response = await client.indices.flush();response = client.indices.flush$resp = $client->indices()->flush();curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_flush"client.indices().flush(f -> f);
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0.
GET _ilm/policy/my_policy
resp = client.ilm.get_lifecycle(
name="my_policy",
)const response = await client.ilm.getLifecycle({
name: "my_policy",
});response = client.ilm.get_lifecycle(
policy: "my_policy"
)$resp = $client->ilm()->getLifecycle([
"policy" => "my_policy",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_ilm/policy/my_policy"client.ilm().getLifecycle(g -> g
.name("my_policy")
);
{
"my_policy": {
"version": 1,
"modified_date": 82392349,
"policy": {
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
},
"in_use_by" : {
"indices" : [],
"data_streams" : [],
"composable_templates" : []
}
}
}
Create or update a lifecycle policy
Generally available; Added in 6.6.0
If the specified policy exists, it is replaced and the policy version is incremented.
NOTE: Only the latest version of the policy is stored, you cannot revert to previous versions.
Required authorization
- Index privileges:
manage - Cluster privileges:
manage_ilm
Query parameters
-
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0. -
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0.
PUT _ilm/policy/my_policy
{
"policy": {
"_meta": {
"description": "used for nginx log",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}resp = client.ilm.put_lifecycle(
name="my_policy",
policy={
"_meta": {
"description": "used for nginx log",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
},
)const response = await client.ilm.putLifecycle({
name: "my_policy",
policy: {
_meta: {
description: "used for nginx log",
project: {
name: "myProject",
department: "myDepartment",
},
},
phases: {
warm: {
min_age: "10d",
actions: {
forcemerge: {
max_num_segments: 1,
},
},
},
delete: {
min_age: "30d",
actions: {
delete: {},
},
},
},
},
});response = client.ilm.put_lifecycle(
policy: "my_policy",
body: {
"policy": {
"_meta": {
"description": "used for nginx log",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}
)$resp = $client->ilm()->putLifecycle([
"policy" => "my_policy",
"body" => [
"policy" => [
"_meta" => [
"description" => "used for nginx log",
"project" => [
"name" => "myProject",
"department" => "myDepartment",
],
],
"phases" => [
"warm" => [
"min_age" => "10d",
"actions" => [
"forcemerge" => [
"max_num_segments" => 1,
],
],
],
"delete" => [
"min_age" => "30d",
"actions" => [
"delete" => new ArrayObject([]),
],
],
],
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"policy":{"_meta":{"description":"used for nginx log","project":{"name":"myProject","department":"myDepartment"}},"phases":{"warm":{"min_age":"10d","actions":{"forcemerge":{"max_num_segments":1}}},"delete":{"min_age":"30d","actions":{"delete":{}}}}}}' "$ELASTICSEARCH_URL/_ilm/policy/my_policy"client.ilm().putLifecycle(p -> p
.name("my_policy")
.policy(po -> po
.phases(ph -> ph
.delete(d -> d
.actions(a -> a
.delete(de -> de)
)
.minAge(m -> m
.time("30d")
)
)
.warm(w -> w
.actions(a -> a
.forcemerge(f -> f
.maxNumSegments(1)
)
)
.minAge(m -> m
.time("10d")
)
)
)
.meta(Map.of("description", JsonData.fromJson("\"used for nginx log\""),"project", JsonData.fromJson("{\"name\":\"myProject\",\"department\":\"myDepartment\"}")))
)
);
{
"policy": {
"_meta": {
"description": "used for nginx log",
"project": {
"name": "myProject",
"department": "myDepartment"
}
},
"phases": {
"warm": {
"min_age": "10d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}{
"acknowledged": true
}Inference
Inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Azure, Google AI Studio or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.
Get Logstash pipelines
Generally available; Added in 7.12.0
All methods and paths for this operation:
Get pipelines that are used for Logstash Central Management.
Required authorization
- Cluster privileges:
manage_logstash_pipelines
GET _logstash/pipeline/my_pipeline
resp = client.logstash.get_pipeline(
id="my_pipeline",
)const response = await client.logstash.getPipeline({
id: "my_pipeline",
});response = client.logstash.get_pipeline(
id: "my_pipeline"
)$resp = $client->logstash()->getPipeline([
"id" => "my_pipeline",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_logstash/pipeline/my_pipeline"client.logstash().getPipeline(g -> g
.id("my_pipeline")
);
{
"my_pipeline": {
"description": "Sample pipeline for illustration purposes",
"last_modified": "2021-01-02T02:50:51.250Z",
"pipeline_metadata": {
"type": "logstash_pipeline",
"version": "1"
},
"username": "elastic",
"pipeline": "input {}\\n filter { grok {} }\\n output {}",
"pipeline_settings": {
"pipeline.workers": 1,
"pipeline.batch.size": 125,
"pipeline.batch.delay": 50,
"queue.type": "memory",
"queue.max_bytes": "1gb",
"queue.checkpoint.writes": 1024
}
}
}
Get overall bucket results
Generally available; Added in 6.1.0
All methods and paths for this operation:
Retrievs overall bucket results that summarize the bucket results of multiple anomaly detection jobs.
The overall_score is calculated by combining the scores of all the
buckets within the overall bucket span. First, the maximum
anomaly_score per anomaly detection job in the overall bucket is
calculated. Then the top_n of those scores are averaged to result in
the overall_score. This means that you can fine-tune the
overall_score so that it is more or less sensitive to the number of
jobs that detect an anomaly at the same time. For example, if you set
top_n to 1, the overall_score is the maximum bucket score in the
overall bucket. Alternatively, if you set top_n to the number of jobs,
the overall_score is high only when all jobs detect anomalies in that
overall bucket. If you set the bucket_span parameter (to a value
greater than its default), the overall_score is the maximum
overall_score of the overall buckets that have a span equal to the
jobs' largest bucket span.
Required authorization
- Cluster privileges:
monitor_ml
Path parameters
-
Identifier for the anomaly detection job. It can be a job identifier, a group name, a comma-separated list of jobs or groups, or a wildcard expression.
You can summarize the bucket results for all anomaly detection jobs by using
_allor by specifying*as the<job_id>.
Query parameters
-
Specifies what to do when the request:
- Contains wildcard expressions and there are no jobs that match.
- Contains the
_allstring or no identifiers and there are no matches. - Contains wildcard expressions and there are only partial matches.
If
true, the request returns an emptyjobsarray when there are no matches and the subset of results when there are partial matches. If this parameter isfalse, the request returns a404status code when there are no matches or only partial matches. -
The span of the overall buckets. Must be greater or equal to the largest bucket span of the specified anomaly detection jobs, which is the default value.
By default, an overall bucket has a span equal to the largest bucket span of the specified anomaly detection jobs. To override that behavior, use the optional
bucket_spanparameter.Values are
-1or0. -
Returns overall buckets with timestamps earlier than this time.
-
If
true, the output excludes interim results. -
Returns overall buckets with overall scores greater than or equal to this value.
-
Returns overall buckets with timestamps after this time.
-
The number of top anomaly detection job bucket scores to be used in the
overall_scorecalculation.
Body
-
Refer to the description for the
allow_no_matchquery parameter.Default value is
true. -
A duration. Units can be
nanos,micros,ms(milliseconds),s(seconds),m(minutes),h(hours) andd(days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. -
Refer to the description for the
exclude_interimquery parameter.Default value is
false. -
Refer to the description for the
top_nquery parameter.Default value is
1.
GET _ml/anomaly_detectors/job-*/results/overall_buckets
{
"overall_score": 80,
"start": "1403532000000"
}resp = client.ml.get_overall_buckets(
job_id="job-*",
overall_score=80,
start="1403532000000",
)const response = await client.ml.getOverallBuckets({
job_id: "job-*",
overall_score: 80,
start: 1403532000000,
});response = client.ml.get_overall_buckets(
job_id: "job-*",
body: {
"overall_score": 80,
"start": "1403532000000"
}
)$resp = $client->ml()->getOverallBuckets([
"job_id" => "job-*",
"body" => [
"overall_score" => 80,
"start" => "1403532000000",
],
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"overall_score":80,"start":"1403532000000"}' "$ELASTICSEARCH_URL/_ml/anomaly_detectors/job-*/results/overall_buckets"client.ml().getOverallBuckets(g -> g
.jobId("job-*")
.overallScore("80")
.start(DateTime.of("1403532000000"))
);
{
"overall_score": 80,
"start": "1403532000000"
}
Get data frame analytics job configuration info
Generally available; Added in 7.3.0
All methods and paths for this operation:
You can get information for multiple data frame analytics jobs in a single API request by using a comma-separated list of data frame analytics jobs or a wildcard expression.
Required authorization
- Cluster privileges:
monitor_ml
Path parameters
-
Identifier for the data frame analytics job. If you do not specify this option, the API returns information for the first hundred data frame analytics jobs.
Query parameters
-
Specifies what to do when the request:
- Contains wildcard expressions and there are no data frame analytics jobs that match.
- Contains the
_allstring or no identifiers and there are no matches. - Contains wildcard expressions and there are only partial matches.
The default value returns an empty data_frame_analytics array when there are no matches and the subset of results when there are partial matches. If this parameter is
false, the request returns a 404 status code when there are no matches or only partial matches. -
Skips the specified number of data frame analytics jobs.
-
Specifies the maximum number of data frame analytics jobs to obtain.
-
Indicates if certain fields should be removed from the configuration on retrieval. This allows the configuration to be in an acceptable format to be retrieved and then added to another cluster.
GET _ml/data_frame/analytics/loganalytics
resp = client.ml.get_data_frame_analytics(
id="loganalytics",
)const response = await client.ml.getDataFrameAnalytics({
id: "loganalytics",
});response = client.ml.get_data_frame_analytics(
id: "loganalytics"
)$resp = $client->ml()->getDataFrameAnalytics([
"id" => "loganalytics",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_ml/data_frame/analytics/loganalytics"client.ml().getDataFrameAnalytics(g -> g
.id("loganalytics")
);
POST /_migration/reindex/my-data-stream/_cancel
resp = client.perform_request(
"POST",
"/_migration/reindex/my-data-stream/_cancel",
)const response = await client.transport.request({
method: "POST",
path: "/_migration/reindex/my-data-stream/_cancel",
});response = client.perform_request(
"POST",
"/_migration/reindex/my-data-stream/_cancel",
{},
)$requestFactory = Psr17FactoryDiscovery::findRequestFactory();
$request = $requestFactory->createRequest(
"POST",
"/_migration/reindex/my-data-stream/_cancel",
);
$resp = $client->sendRequest($request);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_migration/reindex/my-data-stream/_cancel"client.indices().cancelMigrateReindex(c -> c
.index("my-data-stream")
);
Get the shutdown status
Generally available; Added in 7.13.0
All methods and paths for this operation:
Get information about nodes that are ready to be shut down, have shut down preparations still in progress, or have stalled. The API returns status information for each part of the shut down process.
NOTE: This feature is designed for indirect use by Elasticsearch Service, Elastic Cloud Enterprise, and Elastic Cloud on Kubernetes. Direct use is not supported.
If the operator privileges feature is enabled, you must be an operator to use this API.
Required authorization
- Cluster privileges:
manage
GET /_nodes/USpTGYaBSIKbgSUJR2Z9lg/shutdown
resp = client.shutdown.get_node(
node_id="USpTGYaBSIKbgSUJR2Z9lg",
)const response = await client.shutdown.getNode({
node_id: "USpTGYaBSIKbgSUJR2Z9lg",
});response = client.shutdown.get_node(
node_id: "USpTGYaBSIKbgSUJR2Z9lg"
)$resp = $client->shutdown()->getNode([
"node_id" => "USpTGYaBSIKbgSUJR2Z9lg",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_nodes/USpTGYaBSIKbgSUJR2Z9lg/shutdown"client.shutdown().getNode(g -> g
.nodeId("USpTGYaBSIKbgSUJR2Z9lg")
);
{
"nodes": [
{
"node_id": "USpTGYaBSIKbgSUJR2Z9lg",
"type": "RESTART",
"reason": "Demonstrating how the node shutdown API works",
"shutdown_startedmillis": 1624406108685,
"allocation_delay": "10m",
"status": "COMPLETE",
"shard_migration": {
"status": "COMPLETE",
"shard_migrations_remaining": 0,
"explanation": "no shard relocation is necessary for a node restart"
},
"persistent_tasks": {
"status": "COMPLETE"
},
"plugins": {
"status": "COMPLETE"
}
}
]
}Query rules
Query rules enable you to configure per-query rules that are applied at query time to queries that match the specific rule. Query rules are organized into rulesets, collections of query rules that are matched against incoming queries. Query rules are applied using the rule query. If a query matches one or more rules in the ruleset, the query is re-written to apply the rules before searching. This allows pinning documents for only queries that match a specific term.
Stop rollup jobs
Deprecated
Technical preview; Added in 6.3.0
If you try to stop a job that does not exist, an exception occurs. If you try to stop a job that is already stopped, nothing happens.
Since only a stopped job can be deleted, it can be useful to block the API until the indexer has fully stopped.
This is accomplished with the wait_for_completion query parameter, and optionally a timeout. For example:
POST _rollup/job/sensor/_stop?wait_for_completion=true&timeout=10s
The parameter blocks the API call from returning until either the job has moved to STOPPED or the specified time has elapsed. If the specified time elapses without the job moving to STOPPED, a timeout exception occurs.
Required authorization
- Cluster privileges:
manage_rollup
Query parameters
-
If
wait_for_completionistrue, the API blocks for (at maximum) the specified duration while waiting for the job to stop. If more thantimeouttime has passed, the API throws a timeout exception. NOTE: Even if a timeout occurs, the stop request is still processing and eventually moves the job to STOPPED. The timeout simply means the API call itself timed out while waiting for the status change.Values are
-1or0. -
If set to
true, causes the API to block until the indexer state completely stops. If set tofalse, the API returns immediately and the indexer is stopped asynchronously in the background.
POST _rollup/job/sensor/_stop?wait_for_completion=true&timeout=10s
resp = client.rollup.stop_job(
id="sensor",
wait_for_completion=True,
timeout="10s",
)const response = await client.rollup.stopJob({
id: "sensor",
wait_for_completion: "true",
timeout: "10s",
});response = client.rollup.stop_job(
id: "sensor",
wait_for_completion: "true",
timeout: "10s"
)$resp = $client->rollup()->stopJob([
"id" => "sensor",
"wait_for_completion" => "true",
"timeout" => "10s",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_rollup/job/sensor/_stop?wait_for_completion=true&timeout=10s"client.rollup().stopJob(s -> s
.id("sensor")
.timeout(t -> t
.offset(10)
)
.waitForCompletion(true)
);
Get async search results
Generally available; Added in 7.7.0
Retrieve the results of a previously submitted asynchronous search request. If the Elasticsearch security features are enabled, access to the results of a specific async search is restricted to the user or API key that submitted it.
Query parameters
-
The length of time that the async search should be available in the cluster. When not specified, the
keep_aliveset with the corresponding submit async request will be used. Otherwise, it is possible to override the value and extend the validity of the request. When this period expires, the search, if still running, is cancelled. If the search is completed, its saved results are deleted.Values are
-1or0. -
Specify whether aggregation and suggester names should be prefixed by their respective types in the response
-
Specifies to wait for the search to be completed up until the provided timeout. Final results will be returned if available before the timeout expires, otherwise the currently available results will be returned once the timeout expires. By default no timeout is set meaning that the currently available results will be returned without any additional wait.
Values are
-1or0.
GET /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=
resp = client.async_search.get(
id="FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
)const response = await client.asyncSearch.get({
id: "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
});response = client.async_search.get(
id: "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc="
)$resp = $client->asyncSearch()->get([
"id" => "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc="client.asyncSearch().get(g -> g
.id("FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=")
);
{
"id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
"is_partial" : false,
"is_running" : false,
"start_time_in_millis" : 1583945890986,
"expiration_time_in_millis" : 1584377890986,
"completion_time_in_millis" : 1583945903130,
"response" : {
"took" : 12144,
"timed_out" : false,
"num_reduce_phases" : 46,
"_shards" : {
"total" : 562,
"successful" : 188,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 456433,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"sale_date" : {
"buckets" : []
}
}
}
}
Run a search
Generally available
All methods and paths for this operation:
Get search hits that match the query defined in the request.
You can provide search queries using the q query string parameter or the request body.
If both are specified, only the query parameter is used.
If the Elasticsearch security features are enabled, you must have the read index privilege for the target data stream, index, or alias. For cross-cluster search, refer to the documentation about configuring CCS privileges.
To search a point in time (PIT) for an alias, you must have the read index privilege for the alias's data streams or indices.
Search slicing
When paging through a large number of documents, it can be helpful to split the search into multiple slices to consume them independently with the slice and pit properties.
By default the splitting is done first on the shards, then locally on each shard.
The local splitting partitions the shard into contiguous ranges based on Lucene document IDs.
For instance if the number of shards is equal to 2 and you request 4 slices, the slices 0 and 2 are assigned to the first shard and the slices 1 and 3 are assigned to the second shard.
IMPORTANT: The same point-in-time ID should be used for all slices. If different PIT IDs are used, slices can overlap and miss documents. This situation can occur because the splitting criterion is based on Lucene document IDs, which are not stable across changes to the index.
Required authorization
- Index privileges:
read
Path parameters
-
A comma-separated list of data streams, indices, and aliases to search. It supports wildcards (
*). To search all data streams and indices, omit this parameter or use*or_all.
Query parameters
-
If
false, the request returns an error if any wildcard expression, index alias, or_allvalue targets only missing or closed indices. This behavior applies even if the request targets other open indices. For example, a request targetingfoo*,bar*returns an error if an index starts withfoobut no index starts withbar. -
If
trueand there are shard request timeouts or shard failures, the request returns partial results. Iffalse, it returns an error with no partial results.To override the default behavior, you can set the
search.default_allow_partial_resultscluster setting tofalse. -
The analyzer to use for the query string. This parameter can be used only when the
qquery string parameter is specified. -
If
true, wildcard and prefix queries are analyzed. This parameter can be used only when theqquery string parameter is specified. -
The number of shard results that should be reduced at once on the coordinating node. If the potential number of shards in the request can be large, this value should be used as a protection mechanism to reduce the memory overhead per search request.
-
If
true, network round-trips between the coordinating node and the remote clusters are minimized when running cross-cluster search (CCS) requests. -
The default operator for the query string query:
ANDorOR. This parameter can be used only when theqquery string parameter is specified.Values are
and,AND,or, orOR. -
The field to use as a default when no field prefix is given in the query string. This parameter can be used only when the
qquery string parameter is specified. -
A comma-separated list of fields to return as the docvalue representation of a field for each hit.
-
The type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. It supports comma-separated values such as
open,hidden.Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined withopen,closed, orboth.none: Wildcard expressions are not accepted.
Values are
all,open,closed,hidden, ornone. -
If
true, the request returns detailed information about score computation as part of a hit. -
If
true, concrete, expanded or aliased indices will be ignored when frozen. -
If
true, the response includes the score contribution from any named queries.This functionality reruns each named query on every hit in a search response. Typically, this adds a small overhead to a request. However, using computationally expensive named queries on a large number of hits may add significant overhead.
-
If
true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored. This parameter can be used only when theqquery string parameter is specified. -
The number of concurrent shard requests per node that the search runs concurrently. This value should be used to limit the impact of the search on the cluster in order to limit the number of concurrent shard requests.
-
The minimum version of the node that can handle the request Any handling node with a lower version will fail the request.
-
The nodes and shards used for the search. By default, Elasticsearch selects from eligible nodes and shards using adaptive replica selection, accounting for allocation awareness. Valid values are:
_only_localto run the search only on shards on the local node;_localto, if possible, run the search on shards on the local node, or if not, select shards using the default method;_only_nodes:<node-id>,<node-id>to run the search on only the specified nodes IDs, where, if suitable shards exist on more than one selected node, use shards on those nodes using the default method, or if none of the specified nodes are available, select shards from any available node using the default method;_prefer_nodes:<node-id>,<node-id>to if possible, run the search on the specified nodes IDs, or if not, select shards using the default method;_shards:<shard>,<shard>to run the search only on the specified shards;<custom-string>(any string that does not start with_) to route searches with the same<custom-string>to the same shards in the same order.
-
A threshold that enforces a pre-filter roundtrip to prefilter search shards based on query rewriting if the number of shards the search request expands to exceeds the threshold. This filter roundtrip can limit the number of shards significantly if for instance a shard can not match any documents based on its rewrite method (if date filters are mandatory to match but the shard bounds and the query are disjoint). When unspecified, the pre-filter phase is executed if any of these conditions is met:
- The request targets more than 128 shards.
- The request targets one or more read-only index.
- The primary sort of the query targets an indexed field.
-
If
true, the caching of search results is enabled for requests wheresizeis0. It defaults to index level settings. -
A custom value that is used to route operations to a specific shard.
-
The period to retain the search context for scrolling. By default, this value cannot exceed
1d(24 hours). You can change this limit by using thesearch.max_keep_alivecluster-level setting.Values are
-1or0. -
Indicates how distributed term frequencies are calculated for relevance scoring.
Supported values include:
query_then_fetch: Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.dfs_query_then_fetch: Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.
Values are
query_then_fetchordfs_query_then_fetch. -
Specific
tagof the request for logging and statistical purposes. -
A comma-separated list of stored fields to return as part of a hit. If no fields are specified, no stored fields are included in the response. If this field is specified, the
_sourceparameter defaults tofalse. You can pass_source: trueto return both source fields and stored fields in the search response. -
The field to use for suggestions.
-
The suggest mode. This parameter can be used only when the
suggest_fieldandsuggest_textquery string parameters are specified.Supported values include:
missing: Only generate suggestions for terms that are not in the shard.popular: Only suggest terms that occur in more docs on the shard than the original term.always: Suggest any matching suggestions based on terms in the suggest text.
Values are
missing,popular, oralways. -
The number of suggestions to return. This parameter can be used only when the
suggest_fieldandsuggest_textquery string parameters are specified. -
The source text for which the suggestions should be returned. This parameter can be used only when the
suggest_fieldandsuggest_textquery string parameters are specified. -
The maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.
IMPORTANT: Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers. If set to
0(default), the query does not terminate early. -
The period of time to wait for a response from each shard. If no response is received before the timeout expires, the request fails and returns an error. It defaults to no timeout.
Values are
-1or0. -
The number of hits matching the query to count accurately. If
true, the exact number of hits is returned at the cost of some performance. Iffalse, the response does not include the total number of hits matching the query. -
If
true, the request calculates and returns document scores, even if the scores are not used for sorting. -
If
true, aggregation and suggester names are be prefixed by their respective types in the response. -
Indicates whether
hits.totalshould be rendered as an integer or an object in the rest search response. -
If
true, the request returns the document version as part of a hit. -
The source fields that are returned for matching documents. These fields are returned in the
hits._sourceproperty of the search response. Valid values are:trueto return the entire document source.falseto not return the document source.<string>to return the source fields that are specified as a comma-separated list that supports wildcard (*) patterns.
-
A comma-separated list of source fields to exclude from the response. You can also use this parameter to exclude fields from the subset specified in
_source_includesquery parameter. If the_sourceparameter isfalse, this parameter is ignored. -
A comma-separated list of source fields to include in the response. If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the
_source_excludesquery parameter. If the_sourceparameter isfalse, this parameter is ignored. -
If
true, the request returns the sequence number and primary term of the last modification of each hit. -
A query in the Lucene query string syntax. Query parameter searches do not support the full Elasticsearch Query DSL but are handy for testing.
IMPORTANT: This parameter overrides the query parameter in the request body. If both parameters are specified, documents matching the query request body parameter are not returned.
-
The number of hits to return. By default, you cannot page through more than 10,000 hits using the
fromandsizeparameters. To page through more hits, use thesearch_afterparameter. -
The starting document offset, which must be non-negative. By default, you cannot page through more than 10,000 hits using the
fromandsizeparameters. To page through more hits, use thesearch_afterparameter. -
A comma-separated list of
<field>:<direction>pairs.
Body
-
Defines the aggregations that are run as part of the search request.
-
External documentation -
If
true, the request returns detailed information about score computation as part of a hit.Default value is
false. -
Configuration of search extensions defined by Elasticsearch plugins.
-
The starting document offset, which must be non-negative. By default, you cannot page through more than 10,000 hits using the
fromandsizeparameters. To page through more hits, use thesearch_afterparameter.Default value is
0. -
Number of hits matching the query to count accurately. If true, the exact number of hits is returned at the cost of some performance. If false, the response does not include the total number of hits matching the query. Defaults to 10,000 hits.
-
Boost the
_scoreof documents from specified indices. The boost value is the factor by which scores are multiplied. A boost value greater than1.0increases the score. A boost value between0and1.0decreases the score.External documentation -
An array of wildcard (
*) field patterns. The request returns doc values for field names matching these patterns in thehits.fieldsproperty of the response.A reference to a field with formatting instructions on how to return the value
External documentation -
The minimum
_scorefor matching documents. Documents with a lower_scoreare not included in search results and results collected by aggregations. -
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
External documentation -
Set to
trueto return detailed timing information about the execution of individual components in a search request. NOTE: This is a debugging tool and adds significant overhead to search execution.Default value is
false. -
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
External documentation -
Retrieve a script evaluation (based on different fields) for each hit.
-
A field value.
A field value.
-
The number of hits to return, which must not be negative. By default, you cannot page through more than 10,000 hits using the
fromandsizeparameters. To page through more hits, use thesearch_afterproperty.Default value is
10. -
An array of wildcard (
*) field patterns. The request returns values for field names matching these patterns in thehits.fieldsproperty of the response.A reference to a field with formatting instructions on how to return the value
-
The maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.
IMPORTANT: Use with caution. Elasticsearch applies this property to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this property for requests that target data streams with backing indices across multiple data tiers.
If set to
0(default), the query does not terminate early.Default value is
0. -
The period of time to wait for a response from each shard. If no response is received before the timeout expires, the request fails and returns an error. Defaults to no timeout.
-
If
true, calculate and return document scores, even if the scores are not used for sorting.Default value is
false. -
If
true, the request returns the document version as part of a hit.Default value is
false. -
If
true, the request returns sequence number and primary term of the last modification of each hit.External documentation -
The stats groups to associate with the search. Each group maintains a statistics aggregation for its associated searches. You can retrieve these stats using the indices stats API.
GET /my-index-000001/_search?from=40&size=20
{
"query": {
"term": {
"user.id": "kimchy"
}
}
}resp = client.search(
index="my-index-000001",
from="40",
size="20",
query={
"term": {
"user.id": "kimchy"
}
},
)const response = await client.search({
index: "my-index-000001",
from: 40,
size: 20,
query: {
term: {
"user.id": "kimchy",
},
},
});response = client.search(
index: "my-index-000001",
from: "40",
size: "20",
body: {
"query": {
"term": {
"user.id": "kimchy"
}
}
}
)$resp = $client->search([
"index" => "my-index-000001",
"from" => "40",
"size" => "20",
"body" => [
"query" => [
"term" => [
"user.id" => "kimchy",
],
],
],
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"query":{"term":{"user.id":"kimchy"}}}' "$ELASTICSEARCH_URL/my-index-000001/_search?from=40&size=20"client.search(s -> s
.from(40)
.index("my-index-000001")
.query(q -> q
.term(t -> t
.field("user.id")
.value(FieldValue.of("kimchy"))
)
)
.size(20)
,Void.class);
{
"query": {
"term": {
"user.id": "kimchy"
}
}
}{
"size": 100,
"query": {
"match" : {
"title" : "elasticsearch"
}
},
"pit": {
"id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==",
"keep_alive": "1m"
}
}{
"slice": {
"id": 0,
"max": 2
},
"query": {
"match": {
"message": "foo"
}
},
"pit": {
"id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA=="
}
}{
"took": 5,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 20,
"relation": "eq"
},
"max_score": 1.3862942,
"hits": [
{
"_index": "my-index-000001",
"_id": "0",
"_score": 1.3862942,
"_source": {
"@timestamp": "2099-11-15T14:12:12",
"http": {
"request": {
"method": "get"
},
"response": {
"status_code": 200,
"bytes": 1070000
},
"version": "1.1"
},
"source": {
"ip": "127.0.0.1"
},
"message": "GET /search HTTP/1.1 200 1070000",
"user": {
"id": "kimchy"
}
}
}
]
}
}
Get cache statistics
Technical preview; Added in 7.13.0
All methods and paths for this operation:
Get statistics about the shared cache for partially mounted indices.
Required authorization
- Cluster privileges:
manage
GET /_searchable_snapshots/cache/stats
resp = client.searchable_snapshots.cache_stats()const response = await client.searchableSnapshots.cacheStats();response = client.searchable_snapshots.cache_stats$resp = $client->searchableSnapshots()->cacheStats();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_searchable_snapshots/cache/stats"client.searchableSnapshots().cacheStats(c -> c);
{
"nodes" : {
"eerrtBMtQEisohZzxBLUSw" : {
"shared_cache" : {
"reads" : 6051,
"bytes_read_in_bytes" : 5448829,
"writes" : 37,
"bytes_written_in_bytes" : 1208320,
"evictions" : 5,
"num_regions" : 65536,
"size_in_bytes" : 1099511627776,
"region_size_in_bytes" : 16777216
}
}
}
}Snapshot lifecycle management
Snapshot lifecycle management (SLM) APIs enable you to set up policies to automatically take snapshots and control how long they are retained.
Query parameters
-
The period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error. To indicate that the request should never timeout, set it to
-1.Values are
-1or0. -
The period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error. To indicate that the request should never timeout, set it to
-1.Values are
-1or0.
PUT /_slm/policy/daily-snapshots
{
"schedule": "0 30 1 * * ?",
"name": "<daily-snap-{now/d}>",
"repository": "my_repository",
"config": {
"indices": ["data-*", "important"],
"ignore_unavailable": false,
"include_global_state": false
},
"retention": {
"expire_after": "30d",
"min_count": 5,
"max_count": 50
}
}resp = client.slm.put_lifecycle(
policy_id="daily-snapshots",
schedule="0 30 1 * * ?",
name="<daily-snap-{now/d}>",
repository="my_repository",
config={
"indices": [
"data-*",
"important"
],
"ignore_unavailable": False,
"include_global_state": False
},
retention={
"expire_after": "30d",
"min_count": 5,
"max_count": 50
},
)const response = await client.slm.putLifecycle({
policy_id: "daily-snapshots",
schedule: "0 30 1 * * ?",
name: "<daily-snap-{now/d}>",
repository: "my_repository",
config: {
indices: ["data-*", "important"],
ignore_unavailable: false,
include_global_state: false,
},
retention: {
expire_after: "30d",
min_count: 5,
max_count: 50,
},
});response = client.slm.put_lifecycle(
policy_id: "daily-snapshots",
body: {
"schedule": "0 30 1 * * ?",
"name": "<daily-snap-{now/d}>",
"repository": "my_repository",
"config": {
"indices": [
"data-*",
"important"
],
"ignore_unavailable": false,
"include_global_state": false
},
"retention": {
"expire_after": "30d",
"min_count": 5,
"max_count": 50
}
}
)$resp = $client->slm()->putLifecycle([
"policy_id" => "daily-snapshots",
"body" => [
"schedule" => "0 30 1 * * ?",
"name" => "<daily-snap-{now/d}>",
"repository" => "my_repository",
"config" => [
"indices" => array(
"data-*",
"important",
),
"ignore_unavailable" => false,
"include_global_state" => false,
],
"retention" => [
"expire_after" => "30d",
"min_count" => 5,
"max_count" => 50,
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"schedule":"0 30 1 * * ?","name":"<daily-snap-{now/d}>","repository":"my_repository","config":{"indices":["data-*","important"],"ignore_unavailable":false,"include_global_state":false},"retention":{"expire_after":"30d","min_count":5,"max_count":50}}' "$ELASTICSEARCH_URL/_slm/policy/daily-snapshots"client.slm().putLifecycle(p -> p
.config(c -> c
.ignoreUnavailable(false)
.indices(List.of("data-*","important"))
.includeGlobalState(false)
)
.name("<daily-snap-{now/d}>")
.policyId("daily-snapshots")
.repository("my_repository")
.retention(r -> r
.expireAfter(e -> e
.time("30d")
)
.maxCount(50)
.minCount(5)
)
.schedule("0 30 1 * * ?")
);
{
"schedule": "0 30 1 * * ?",
"name": "<daily-snap-{now/d}>",
"repository": "my_repository",
"config": {
"indices": ["data-*", "important"],
"ignore_unavailable": false,
"include_global_state": false
},
"retention": {
"expire_after": "30d",
"min_count": 5,
"max_count": 50
}
}{
"schedule": "1h",
"name": "<hourly-snap-{now/d}>",
"repository": "my_repository",
"config": {
"indices": ["data-*", "important"]
}
}Query parameters
-
The period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error. To indicate that the request should never timeout, set it to
-1.Values are
-1or0. -
The period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error. To indicate that the request should never timeout, set it to
-1.Values are
-1or0.
POST _slm/start
resp = client.slm.start()const response = await client.slm.start();response = client.slm.start$resp = $client->slm()->start();curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_slm/start"client.slm().start(s -> s);
{
"acknowledged": true
}
Delete an async SQL search
Generally available; Added in 7.15.0
Delete an async SQL search or a stored synchronous SQL search. If the search is still running, the API cancels it.
If the Elasticsearch security features are enabled, only the following users can use this API to delete a search:
- Users with the
cancel_taskcluster privilege. - The user who first submitted the search.
Required authorization
- Cluster privileges:
cancel_task
DELETE _sql/async/delete/FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI=
resp = client.sql.delete_async(
id="FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI=",
)const response = await client.sql.deleteAsync({
id: "FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI=",
});response = client.sql.delete_async(
id: "FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI="
)$resp = $client->sql()->deleteAsync([
"id" => "FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI=",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_sql/async/delete/FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI="client.sql().deleteAsync(d -> d
.id("FmdMX2pIang3UWhLRU5QS0lqdlppYncaMUpYQ05oSkpTc3kwZ21EdC1tbFJXQToxOTI=")
);
Get task information
Technical preview; Added in 5.0.0
Get information about a task currently running in the cluster.
WARNING: The task management API is new and should still be considered a beta feature. The API may change in ways that are not backwards compatible.
If the task identifier is not found, a 404 response code indicates that there are no resources that match the request.
Required authorization
- Cluster privileges:
monitor
GET _tasks?detailed=true&actions=*/delete/byquery
resp = client.tasks.list(
detailed=True,
actions="*/delete/byquery",
)const response = await client.tasks.list({
detailed: "true",
actions: "*/delete/byquery",
});response = client.tasks.list(
detailed: "true",
actions: "*/delete/byquery"
)$resp = $client->tasks()->list([
"detailed" => "true",
"actions" => "*/delete/byquery",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_tasks?detailed=true&actions=*/delete/byquery"{
"nodes" : {
"oTUltX4IQMOUUVeiohTt8A" : {
"name" : "H5dfFeA",
"transport_address" : "127.0.0.1:9300",
"host" : "127.0.0.1",
"ip" : "127.0.0.1:9300",
"tasks" : {
"oTUltX4IQMOUUVeiohTt8A:124" : {
"node" : "oTUltX4IQMOUUVeiohTt8A",
"id" : 124,
"type" : "direct",
"action" : "cluster:monitor/tasks/lists[n]",
"start_time_in_millis" : 1458585884904,
"running_time_in_nanos" : 47402,
"cancellable" : false,
"parent_task_id" : "oTUltX4IQMOUUVeiohTt8A:123"
},
"oTUltX4IQMOUUVeiohTt8A:123" : {
"node" : "oTUltX4IQMOUUVeiohTt8A",
"id" : 123,
"type" : "transport",
"action" : "cluster:monitor/tasks/lists",
"start_time_in_millis" : 1458585884904,
"running_time_in_nanos" : 236042,
"cancellable" : false
}
}
}
}
}{
"nodes" : {
"r1A2WoRbTwKZ516z6NEs5A" : {
"name" : "r1A2WoR",
"transport_address" : "127.0.0.1:9300",
"host" : "127.0.0.1",
"ip" : "127.0.0.1:9300",
"attributes" : {
"testattr" : "test",
"portsfile" : "true"
},
"tasks" : {
"r1A2WoRbTwKZ516z6NEs5A:36619" : {
"node" : "r1A2WoRbTwKZ516z6NEs5A",
"id" : 36619,
"type" : "transport",
"action" : "indices:data/write/delete/byquery",
"status" : {
"total" : 6154,
"updated" : 0,
"created" : 0,
"deleted" : 3500,
"batches" : 36,
"version_conflicts" : 0,
"noops" : 0,
"retries": 0,
"throttled_millis": 0
},
"description" : ""
}
}
}
}
}
Create a transform
Generally available; Added in 7.2.0
Creates a transform.
A transform copies data from source indices, transforms it, and persists it into an entity-centric destination index. You can also think of the destination index as a two-dimensional tabular data structure (known as a data frame). The ID for each document in the data frame is generated from a hash of the entity, so there is a unique row per entity.
You must choose either the latest or pivot method for your transform; you cannot use both in a single transform. If
you choose to use the pivot method for your transform, the entities are defined by the set of group_by fields in
the pivot object. If you choose to use the latest method, the entities are defined by the unique_key field values
in the latest object.
You must have create_index, index, and read privileges on the destination index and read and
view_index_metadata privileges on the source indices. When Elasticsearch security features are enabled, the
transform remembers which roles the user that created it had at the time of creation and uses those same roles. If
those roles do not have the required privileges on the source and destination indices, the transform fails when it
attempts unauthorized operations.
NOTE: You must use Kibana or this API to create a transform. Do not add a transform directly into any
.transform-internal* indices using the Elasticsearch index API. If Elasticsearch security features are enabled, do
not give users any privileges on .transform-internal* indices. If you used transforms prior to 7.5, also do not
give users any privileges on .data-frame-internal* indices.
Required authorization
- Index privileges:
create_index,read,index,view_index_metadata - Cluster privileges:
manage_transform
Path parameters
-
Identifier for the transform. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It has a 64 character limit and must start and end with alphanumeric characters.
Query parameters
-
When the transform is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the source indices and a check that the destination index is not part of the source index pattern. You can use this parameter to skip the checks, for example when the source index does not exist until after the transform is created. The validations are always run when you start the transform, however, with the exception of privilege checks.
-
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0.
PUT _transform/ecommerce_transform1
{
"source": {
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
"pivot": {
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": true
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
"description": "Maximum priced ecommerce data by customer_id in Asia",
"dest": {
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
"frequency": "5m",
"sync": {
"time": {
"field": "order_date",
"delay": "60s"
}
},
"retention_policy": {
"time": {
"field": "order_date",
"max_age": "30d"
}
}
}resp = client.transform.put_transform(
transform_id="ecommerce_transform1",
source={
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
pivot={
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": True
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
description="Maximum priced ecommerce data by customer_id in Asia",
dest={
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
frequency="5m",
sync={
"time": {
"field": "order_date",
"delay": "60s"
}
},
retention_policy={
"time": {
"field": "order_date",
"max_age": "30d"
}
},
)const response = await client.transform.putTransform({
transform_id: "ecommerce_transform1",
source: {
index: "kibana_sample_data_ecommerce",
query: {
term: {
"geoip.continent_name": {
value: "Asia",
},
},
},
},
pivot: {
group_by: {
customer_id: {
terms: {
field: "customer_id",
missing_bucket: true,
},
},
},
aggregations: {
max_price: {
max: {
field: "taxful_total_price",
},
},
},
},
description: "Maximum priced ecommerce data by customer_id in Asia",
dest: {
index: "kibana_sample_data_ecommerce_transform1",
pipeline: "add_timestamp_pipeline",
},
frequency: "5m",
sync: {
time: {
field: "order_date",
delay: "60s",
},
},
retention_policy: {
time: {
field: "order_date",
max_age: "30d",
},
},
});response = client.transform.put_transform(
transform_id: "ecommerce_transform1",
body: {
"source": {
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
"pivot": {
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": true
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
"description": "Maximum priced ecommerce data by customer_id in Asia",
"dest": {
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
"frequency": "5m",
"sync": {
"time": {
"field": "order_date",
"delay": "60s"
}
},
"retention_policy": {
"time": {
"field": "order_date",
"max_age": "30d"
}
}
}
)$resp = $client->transform()->putTransform([
"transform_id" => "ecommerce_transform1",
"body" => [
"source" => [
"index" => "kibana_sample_data_ecommerce",
"query" => [
"term" => [
"geoip.continent_name" => [
"value" => "Asia",
],
],
],
],
"pivot" => [
"group_by" => [
"customer_id" => [
"terms" => [
"field" => "customer_id",
"missing_bucket" => true,
],
],
],
"aggregations" => [
"max_price" => [
"max" => [
"field" => "taxful_total_price",
],
],
],
],
"description" => "Maximum priced ecommerce data by customer_id in Asia",
"dest" => [
"index" => "kibana_sample_data_ecommerce_transform1",
"pipeline" => "add_timestamp_pipeline",
],
"frequency" => "5m",
"sync" => [
"time" => [
"field" => "order_date",
"delay" => "60s",
],
],
"retention_policy" => [
"time" => [
"field" => "order_date",
"max_age" => "30d",
],
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"source":{"index":"kibana_sample_data_ecommerce","query":{"term":{"geoip.continent_name":{"value":"Asia"}}}},"pivot":{"group_by":{"customer_id":{"terms":{"field":"customer_id","missing_bucket":true}}},"aggregations":{"max_price":{"max":{"field":"taxful_total_price"}}}},"description":"Maximum priced ecommerce data by customer_id in Asia","dest":{"index":"kibana_sample_data_ecommerce_transform1","pipeline":"add_timestamp_pipeline"},"frequency":"5m","sync":{"time":{"field":"order_date","delay":"60s"}},"retention_policy":{"time":{"field":"order_date","max_age":"30d"}}}' "$ELASTICSEARCH_URL/_transform/ecommerce_transform1"client.transform().putTransform(p -> p
.description("Maximum priced ecommerce data by customer_id in Asia")
.dest(d -> d
.index("kibana_sample_data_ecommerce_transform1")
.pipeline("add_timestamp_pipeline")
)
.frequency(f -> f
.time("5m")
)
.pivot(pi -> pi
.aggregations("max_price", a -> a
.max(m -> m
.field("taxful_total_price")
)
)
.groupBy("customer_id", g -> g
.terms(t -> t
.field("customer_id")
.missingBucket(true)
)
)
)
.retentionPolicy(r -> r
.time(t -> t
.field("order_date")
.maxAge(m -> m
.time("30d")
)
)
)
.source(s -> s
.index("kibana_sample_data_ecommerce")
.query(q -> q
.term(te -> te
.field("geoip.continent_name")
.value(FieldValue.of("Asia"))
)
)
)
.sync(sy -> sy
.time(ti -> ti
.delay(d -> d
.time("60s")
)
.field("order_date")
)
)
.transformId("ecommerce_transform1")
);
{
"source": {
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
"pivot": {
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": true
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
"description": "Maximum priced ecommerce data by customer_id in Asia",
"dest": {
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
"frequency": "5m",
"sync": {
"time": {
"field": "order_date",
"delay": "60s"
}
},
"retention_policy": {
"time": {
"field": "order_date",
"max_age": "30d"
}
}
}{
"source": {
"index": "kibana_sample_data_ecommerce"
},
"latest": {
"unique_key": [
"customer_id"
],
"sort": "order_date"
},
"description": "Latest order for each customer",
"dest": {
"index": "kibana_sample_data_ecommerce_transform2"
},
"frequency": "5m",
"sync": {
"time": {
"field": "order_date",
"delay": "60s"
}
}
}{
"acknowledged": true
}Watcher
You can use Watcher to watch for changes or anomalies in your data and perform the necessary actions in response.
Acknowledge a watch
Generally available
All methods and paths for this operation:
Acknowledging a watch enables you to manually throttle the execution of the watch's actions.
The acknowledgement state of an action is stored in the status.actions.<id>.ack.state structure.
IMPORTANT: If the specified watch is currently being executed, this API will return an error The reason for this behavior is to prevent overwriting the watch status from a watch execution.
Acknowledging an action throttles further executions of that action until its ack.state is reset to awaits_successful_execution.
This happens when the condition of the watch is not met (the condition evaluates to false).
Required authorization
- Cluster privileges:
manage_watcher
POST _watcher/watch/my_watch/_ack
resp = client.watcher.ack_watch(
watch_id="my_watch",
)const response = await client.watcher.ackWatch({
watch_id: "my_watch",
});response = client.watcher.ack_watch(
watch_id: "my_watch"
)$resp = $client->watcher()->ackWatch([
"watch_id" => "my_watch",
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_watcher/watch/my_watch/_ack"client.watcher().ackWatch(a -> a
.watchId("my_watch")
);
{
"status": {
"state": {
"active": true,
"timestamp": "2015-05-26T18:04:27.723Z"
},
"last_checked": "2015-05-26T18:04:27.753Z",
"last_met_condition": "2015-05-26T18:04:27.763Z",
"actions": {
"test_index": {
"ack" : {
"timestamp": "2015-05-26T18:04:27.713Z",
"state": "acked"
},
"last_execution" : {
"timestamp": "2015-05-25T18:04:27.733Z",
"successful": true
},
"last_successful_execution" : {
"timestamp": "2015-05-25T18:04:27.773Z",
"successful": true
}
}
},
"execution_state": "executed",
"version": 2
}
}