Elasticsearch APIs use key-based authentication. You must create an API key and use the encoded value in the request header. For example:
curl -X GET "${ES_URL}/_cat/indices?v=true" \
-H "Authorization: ApiKey ${API_KEY}"
For more information about where to find API keys for the Elasticsearch endpoint (${ES_URL}) for a project, go to Get started with Elasticsearch Serverless.
DELETE _application/analytics/my_analytics_collection/
resp = client.search_application.delete_behavioral_analytics(
name="my_analytics_collection",
)const response = await client.searchApplication.deleteBehavioralAnalytics({
name: "my_analytics_collection",
});response = client.search_application.delete_behavioral_analytics(
name: "my_analytics_collection"
)$resp = $client->searchApplication()->deleteBehavioralAnalytics([
"name" => "my_analytics_collection",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_application/analytics/my_analytics_collection/"client.searchApplication().deleteBehavioralAnalytics(d -> d
.name("my_analytics_collection")
);
All methods and paths for this operation:
Get quick access to a document count for a data stream, an index, or an entire cluster. The document count only includes live documents, not deleted documents which have not yet been removed by the merge process.
IMPORTANT: CAT APIs are only intended for human consumption using the command line or Kibana console. They are not intended for use by applications. For application consumption, use the count API.
readA comma-separated list of data streams, indices, and aliases used to limit the request.
It supports wildcards (*).
To target all data streams and indices, omit this parameter or use * or _all.
GET /_cat/count/my-index-000001?v=true&format=json
resp = client.cat.count(
index="my-index-000001",
v=True,
format="json",
)const response = await client.cat.count({
index: "my-index-000001",
v: "true",
format: "json",
});response = client.cat.count(
index: "my-index-000001",
v: "true",
format: "json"
)$resp = $client->cat()->count([
"index" => "my-index-000001",
"v" => "true",
"format" => "json",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_cat/count/my-index-000001?v=true&format=json"client.cat().count();
[
{
"epoch": "1475868259",
"timestamp": "15:24:20",
"count": "120"
}
][
{
"epoch": "1475868259",
"timestamp": "15:24:20",
"count": "121"
}
]curl \
--request GET 'http://api.example.com/_cat' \
--header "Authorization: $API_KEY"curl \
--request HEAD 'http://api.example.com/' \
--header "Authorization: $API_KEY"The connector and sync jobs APIs provide a convenient way to create and manage Elastic connectors and sync jobs in an internal index.
Connectors are Elasticsearch integrations for syncing content from third-party data sources, which can be deployed on Elastic Cloud or hosted on your own infrastructure.
This API provides an alternative to relying solely on Kibana UI for connector and sync job management. The API comes with a set of validations and assertions to ensure that the state representation in the internal index remains valid.
This API requires the manage_connector privilege or, for read-only endpoints, the monitor_connector privilege.
Update the last_seen field in the connector and set it to the current timestamp.
PUT _connector/my-connector/_check_in
resp = client.connector.check_in(
connector_id="my-connector",
)const response = await client.connector.checkIn({
connector_id: "my-connector",
});response = client.connector.check_in(
connector_id: "my-connector"
)$resp = $client->connector()->checkIn([
"connector_id" => "my-connector",
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/my-connector/_check_in"client.connector().checkIn(c -> c
.connectorId("my-connector")
);
{
"result": "updated"
}All methods and paths for this operation:
The unique identifier of the connector to be created or updated. ID is auto-generated if not provided.
PUT _connector/my-connector
{
"index_name": "search-google-drive",
"name": "My Connector",
"service_type": "google_drive"
}resp = client.connector.put(
connector_id="my-connector",
index_name="search-google-drive",
name="My Connector",
service_type="google_drive",
)const response = await client.connector.put({
connector_id: "my-connector",
index_name: "search-google-drive",
name: "My Connector",
service_type: "google_drive",
});response = client.connector.put(
connector_id: "my-connector",
body: {
"index_name": "search-google-drive",
"name": "My Connector",
"service_type": "google_drive"
}
)$resp = $client->connector()->put([
"connector_id" => "my-connector",
"body" => [
"index_name" => "search-google-drive",
"name" => "My Connector",
"service_type" => "google_drive",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"index_name":"search-google-drive","name":"My Connector","service_type":"google_drive"}' "$ELASTICSEARCH_URL/_connector/my-connector"client.connector().put(p -> p
.connectorId("my-connector")
.indexName("search-google-drive")
.name("My Connector")
.serviceType("google_drive")
);
{
"index_name": "search-google-drive",
"name": "My Connector",
"service_type": "google_drive"
}{
"index_name": "search-google-drive",
"name": "My Connector",
"description": "My Connector to sync data to Elastic index from Google Drive",
"service_type": "google_drive",
"language": "english"
}{
"result": "created",
"id": "my-connector"
}Removes a connector and associated sync jobs. This is a destructive action that is not recoverable. NOTE: This action doesn’t delete any API keys, ingest pipelines, or data indices associated with the connector. These need to be removed manually.
DELETE _connector/my-connector-id&delete_sync_jobs=true
resp = client.connector.delete(
connector_id="my-connector-id&delete_sync_jobs=true",
)const response = await client.connector.delete({
connector_id: "my-connector-id&delete_sync_jobs=true",
});response = client.connector.delete(
connector_id: "my-connector-id&delete_sync_jobs=true"
)$resp = $client->connector()->delete([
"connector_id" => "my-connector-id&delete_sync_jobs=true",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/my-connector-id&delete_sync_jobs=true"client.connector().delete(d -> d
.connectorId("my-connector-id&delete_sync_jobs=true")
);
{
"acknowledged": true
}Starting offset (default: 0)
Specifies a max number of results to get
A comma-separated list of connector index names to fetch connector documents for
A comma-separated list of connector names to fetch connector documents for
A comma-separated list of connector service types to fetch connector documents for
A flag to indicate if the desired connector should be fetched, even if it was soft-deleted.
A wildcard query string that filters connectors with matching name, description or index name
GET _connector
resp = client.connector.list()const response = await client.connector.list();response = client.connector.list$resp = $client->connector()->list();curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector"client.connector().list(l -> l);
Cancel a connector sync job, which sets the status to cancelling and updates cancellation_requested_at to the current time.
The connector service is then responsible for setting the status of connector sync jobs to cancelled.
PUT _connector/_sync_job/my-connector-sync-job-id/_cancel
resp = client.connector.sync_job_cancel(
connector_sync_job_id="my-connector-sync-job-id",
)const response = await client.connector.syncJobCancel({
connector_sync_job_id: "my-connector-sync-job-id",
});response = client.connector.sync_job_cancel(
connector_sync_job_id: "my-connector-sync-job-id"
)$resp = $client->connector()->syncJobCancel([
"connector_sync_job_id" => "my-connector-sync-job-id",
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/_sync_job/my-connector-sync-job-id/_cancel"client.connector().syncJobCancel(s -> s
.connectorSyncJobId("my-connector-sync-job-id")
);
GET _connector/_sync_job/my-connector-sync-job
resp = client.connector.sync_job_get(
connector_sync_job_id="my-connector-sync-job",
)const response = await client.connector.syncJobGet({
connector_sync_job_id: "my-connector-sync-job",
});response = client.connector.sync_job_get(
connector_sync_job_id: "my-connector-sync-job"
)$resp = $client->connector()->syncJobGet([
"connector_sync_job_id" => "my-connector-sync-job",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/_sync_job/my-connector-sync-job"client.connector().syncJobGet(s -> s
.connectorSyncJobId("my-connector-sync-job")
);
Get information about all stored connector sync jobs listed by their creation date in ascending order.
Starting offset (default: 0)
Specifies a max number of results to get
A sync job status to fetch connector sync jobs for
Values are canceling, canceled, completed, error, in_progress, pending, or suspended.
A connector id to fetch connector sync jobs for
A comma-separated list of job types to fetch the sync jobs for
Supported values include: full, incremental, access_control
Values are full, incremental, or access_control.
GET _connector/_sync_job?connector_id=my-connector-id&size=1
resp = client.connector.sync_job_list(
connector_id="my-connector-id",
size="1",
)const response = await client.connector.syncJobList({
connector_id: "my-connector-id",
size: 1,
});response = client.connector.sync_job_list(
connector_id: "my-connector-id",
size: "1"
)$resp = $client->connector()->syncJobList([
"connector_id" => "my-connector-id",
"size" => "1",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_connector/_sync_job?connector_id=my-connector-id&size=1"client.connector().syncJobList(s -> s
.connectorId("my-connector-id")
.size(1)
);
Create a connector sync job document in the internal index and initialize its counters and timestamps with default values.
POST _connector/_sync_job
{
"id": "connector-id",
"job_type": "full",
"trigger_method": "on_demand"
}resp = client.connector.sync_job_post(
id="connector-id",
job_type="full",
trigger_method="on_demand",
)const response = await client.connector.syncJobPost({
id: "connector-id",
job_type: "full",
trigger_method: "on_demand",
});response = client.connector.sync_job_post(
body: {
"id": "connector-id",
"job_type": "full",
"trigger_method": "on_demand"
}
)$resp = $client->connector()->syncJobPost([
"body" => [
"id" => "connector-id",
"job_type" => "full",
"trigger_method" => "on_demand",
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"id":"connector-id","job_type":"full","trigger_method":"on_demand"}' "$ELASTICSEARCH_URL/_connector/_sync_job"client.connector().syncJobPost(s -> s
.id("connector-id")
.jobType(SyncJobType.Full)
.triggerMethod(SyncJobTriggerMethod.OnDemand)
);
{
"id": "connector-id",
"job_type": "full",
"trigger_method": "on_demand"
}curl \
--request PUT 'http://api.example.com/_connector/{connector_id}/_filtering/_activate' \
--header "Authorization: $API_KEY"Update the configuration field in the connector document.
PUT _connector/my-spo-connector/_configuration
{
"values": {
"tenant_id": "my-tenant-id",
"tenant_name": "my-sharepoint-site",
"client_id": "foo",
"secret_value": "bar",
"site_collections": "*"
}
}resp = client.connector.update_configuration(
connector_id="my-spo-connector",
values={
"tenant_id": "my-tenant-id",
"tenant_name": "my-sharepoint-site",
"client_id": "foo",
"secret_value": "bar",
"site_collections": "*"
},
)const response = await client.connector.updateConfiguration({
connector_id: "my-spo-connector",
values: {
tenant_id: "my-tenant-id",
tenant_name: "my-sharepoint-site",
client_id: "foo",
secret_value: "bar",
site_collections: "*",
},
});response = client.connector.update_configuration(
connector_id: "my-spo-connector",
body: {
"values": {
"tenant_id": "my-tenant-id",
"tenant_name": "my-sharepoint-site",
"client_id": "foo",
"secret_value": "bar",
"site_collections": "*"
}
}
)$resp = $client->connector()->updateConfiguration([
"connector_id" => "my-spo-connector",
"body" => [
"values" => [
"tenant_id" => "my-tenant-id",
"tenant_name" => "my-sharepoint-site",
"client_id" => "foo",
"secret_value" => "bar",
"site_collections" => "*",
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"values":{"tenant_id":"my-tenant-id","tenant_name":"my-sharepoint-site","client_id":"foo","secret_value":"bar","site_collections":"*"}}' "$ELASTICSEARCH_URL/_connector/my-spo-connector/_configuration"client.connector().updateConfiguration(u -> u
.connectorId("my-spo-connector")
.values(Map.of("tenant_id", JsonData.fromJson("\"my-tenant-id\""),"tenant_name", JsonData.fromJson("\"my-sharepoint-site\""),"secret_value", JsonData.fromJson("\"bar\""),"client_id", JsonData.fromJson("\"foo\""),"site_collections", JsonData.fromJson("\"*\"")))
);
{
"values": {
"tenant_id": "my-tenant-id",
"tenant_name": "my-sharepoint-site",
"client_id": "foo",
"secret_value": "bar",
"site_collections": "*"
}
}{
"values": {
"secret_value": "foo-bar"
}
}{
"result": "updated"
}Set the error field for the connector. If the error provided in the request body is non-null, the connector’s status is updated to error. Otherwise, if the error is reset to null, the connector status is updated to connected.
PUT _connector/my-connector/_error
{
"error": "Houston, we have a problem!"
}resp = client.connector.update_error(
connector_id="my-connector",
error="Houston, we have a problem!",
)const response = await client.connector.updateError({
connector_id: "my-connector",
error: "Houston, we have a problem!",
});response = client.connector.update_error(
connector_id: "my-connector",
body: {
"error": "Houston, we have a problem!"
}
)$resp = $client->connector()->updateError([
"connector_id" => "my-connector",
"body" => [
"error" => "Houston, we have a problem!",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"error":"Houston, we have a problem!"}' "$ELASTICSEARCH_URL/_connector/my-connector/_error"client.connector().updateError(u -> u
.connectorId("my-connector")
.error("Houston, we have a problem!")
);
{
"error": "Houston, we have a problem!"
}{
"result": "updated"
}Update the draft filtering configuration of a connector and marks the draft validation state as edited. The filtering draft is activated once validated by the running Elastic connector service. The filtering property is used to configure sync rules (both basic and advanced) for a connector.
PUT _connector/my-g-drive-connector/_filtering
{
"rules": [
{
"field": "file_extension",
"id": "exclude-txt-files",
"order": 0,
"policy": "exclude",
"rule": "equals",
"value": "txt"
},
{
"field": "_",
"id": "DEFAULT",
"order": 1,
"policy": "include",
"rule": "regex",
"value": ".*"
}
]
}resp = client.connector.update_filtering(
connector_id="my-g-drive-connector",
rules=[
{
"field": "file_extension",
"id": "exclude-txt-files",
"order": 0,
"policy": "exclude",
"rule": "equals",
"value": "txt"
},
{
"field": "_",
"id": "DEFAULT",
"order": 1,
"policy": "include",
"rule": "regex",
"value": ".*"
}
],
)const response = await client.connector.updateFiltering({
connector_id: "my-g-drive-connector",
rules: [
{
field: "file_extension",
id: "exclude-txt-files",
order: 0,
policy: "exclude",
rule: "equals",
value: "txt",
},
{
field: "_",
id: "DEFAULT",
order: 1,
policy: "include",
rule: "regex",
value: ".*",
},
],
});response = client.connector.update_filtering(
connector_id: "my-g-drive-connector",
body: {
"rules": [
{
"field": "file_extension",
"id": "exclude-txt-files",
"order": 0,
"policy": "exclude",
"rule": "equals",
"value": "txt"
},
{
"field": "_",
"id": "DEFAULT",
"order": 1,
"policy": "include",
"rule": "regex",
"value": ".*"
}
]
}
)$resp = $client->connector()->updateFiltering([
"connector_id" => "my-g-drive-connector",
"body" => [
"rules" => array(
[
"field" => "file_extension",
"id" => "exclude-txt-files",
"order" => 0,
"policy" => "exclude",
"rule" => "equals",
"value" => "txt",
],
[
"field" => "_",
"id" => "DEFAULT",
"order" => 1,
"policy" => "include",
"rule" => "regex",
"value" => ".*",
],
),
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"rules":[{"field":"file_extension","id":"exclude-txt-files","order":0,"policy":"exclude","rule":"equals","value":"txt"},{"field":"_","id":"DEFAULT","order":1,"policy":"include","rule":"regex","value":".*"}]}' "$ELASTICSEARCH_URL/_connector/my-g-drive-connector/_filtering"client.connector().updateFiltering(u -> u
.connectorId("my-g-drive-connector")
.rules(List.of(FilteringRule.of(f -> f
.field("file_extension")
.id("exclude-txt-files")
.order(0)
.policy(FilteringPolicy.Exclude)
.rule(FilteringRuleRule.Equals)
.value("txt")),FilteringRule.of(f -> f
.field("_")
.id("DEFAULT")
.order(1)
.policy(FilteringPolicy.Include)
.rule(FilteringRuleRule.Regex)
.value(".*"))))
);
{
"rules": [
{
"field": "file_extension",
"id": "exclude-txt-files",
"order": 0,
"policy": "exclude",
"rule": "equals",
"value": "txt"
},
{
"field": "_",
"id": "DEFAULT",
"order": 1,
"policy": "include",
"rule": "regex",
"value": ".*"
}
]
}{
"advanced_snippet": {
"value": [{
"tables": [
"users",
"orders"
],
"query": "SELECT users.id AS id, orders.order_id AS order_id FROM users JOIN orders ON users.id = orders.user_id"
}]
}
}{
"result": "updated"
}Update the draft filtering validation info for a connector.
curl \
--request PUT 'http://api.example.com/_connector/{connector_id}/_filtering/_validation' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '{"validation":{"errors":[{"ids":["string"],"messages":["string"]}],"state":"edited"}}'Update the index_name field of a connector, specifying the index where the data ingested by the connector is stored.
PUT _connector/my-connector/_index_name
{
"index_name": "data-from-my-google-drive"
}resp = client.connector.update_index_name(
connector_id="my-connector",
index_name="data-from-my-google-drive",
)const response = await client.connector.updateIndexName({
connector_id: "my-connector",
index_name: "data-from-my-google-drive",
});response = client.connector.update_index_name(
connector_id: "my-connector",
body: {
"index_name": "data-from-my-google-drive"
}
)$resp = $client->connector()->updateIndexName([
"connector_id" => "my-connector",
"body" => [
"index_name" => "data-from-my-google-drive",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"index_name":"data-from-my-google-drive"}' "$ELASTICSEARCH_URL/_connector/my-connector/_index_name"client.connector().updateIndexName(u -> u
.connectorId("my-connector")
.indexName("data-from-my-google-drive")
);
{
"index_name": "data-from-my-google-drive"
}{
"result": "updated"
}PUT _connector/my-connector/_name
{
"name": "Custom connector",
"description": "This is my customized connector"
}resp = client.connector.update_name(
connector_id="my-connector",
name="Custom connector",
description="This is my customized connector",
)const response = await client.connector.updateName({
connector_id: "my-connector",
name: "Custom connector",
description: "This is my customized connector",
});response = client.connector.update_name(
connector_id: "my-connector",
body: {
"name": "Custom connector",
"description": "This is my customized connector"
}
)$resp = $client->connector()->updateName([
"connector_id" => "my-connector",
"body" => [
"name" => "Custom connector",
"description" => "This is my customized connector",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"name":"Custom connector","description":"This is my customized connector"}' "$ELASTICSEARCH_URL/_connector/my-connector/_name"client.connector().updateName(u -> u
.connectorId("my-connector")
.description("This is my customized connector")
.name("Custom connector")
);
{
"name": "Custom connector",
"description": "This is my customized connector"
}{
"result": "updated"
}PUT _connector/my-connector/_scheduling
{
"scheduling": {
"access_control": {
"enabled": true,
"interval": "0 10 0 * * ?"
},
"full": {
"enabled": true,
"interval": "0 20 0 * * ?"
},
"incremental": {
"enabled": false,
"interval": "0 30 0 * * ?"
}
}
}resp = client.connector.update_scheduling(
connector_id="my-connector",
scheduling={
"access_control": {
"enabled": True,
"interval": "0 10 0 * * ?"
},
"full": {
"enabled": True,
"interval": "0 20 0 * * ?"
},
"incremental": {
"enabled": False,
"interval": "0 30 0 * * ?"
}
},
)const response = await client.connector.updateScheduling({
connector_id: "my-connector",
scheduling: {
access_control: {
enabled: true,
interval: "0 10 0 * * ?",
},
full: {
enabled: true,
interval: "0 20 0 * * ?",
},
incremental: {
enabled: false,
interval: "0 30 0 * * ?",
},
},
});response = client.connector.update_scheduling(
connector_id: "my-connector",
body: {
"scheduling": {
"access_control": {
"enabled": true,
"interval": "0 10 0 * * ?"
},
"full": {
"enabled": true,
"interval": "0 20 0 * * ?"
},
"incremental": {
"enabled": false,
"interval": "0 30 0 * * ?"
}
}
}
)$resp = $client->connector()->updateScheduling([
"connector_id" => "my-connector",
"body" => [
"scheduling" => [
"access_control" => [
"enabled" => true,
"interval" => "0 10 0 * * ?",
],
"full" => [
"enabled" => true,
"interval" => "0 20 0 * * ?",
],
"incremental" => [
"enabled" => false,
"interval" => "0 30 0 * * ?",
],
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"scheduling":{"access_control":{"enabled":true,"interval":"0 10 0 * * ?"},"full":{"enabled":true,"interval":"0 20 0 * * ?"},"incremental":{"enabled":false,"interval":"0 30 0 * * ?"}}}' "$ELASTICSEARCH_URL/_connector/my-connector/_scheduling"client.connector().updateScheduling(u -> u
.connectorId("my-connector")
.scheduling(s -> s
.accessControl(a -> a
.enabled(true)
.interval("0 10 0 * * ?")
)
.full(f -> f
.enabled(true)
.interval("0 20 0 * * ?")
)
.incremental(i -> i
.enabled(false)
.interval("0 30 0 * * ?")
)
)
);
{
"scheduling": {
"access_control": {
"enabled": true,
"interval": "0 10 0 * * ?"
},
"full": {
"enabled": true,
"interval": "0 20 0 * * ?"
},
"incremental": {
"enabled": false,
"interval": "0 30 0 * * ?"
}
}
}{
"scheduling": {
"full": {
"enabled": true,
"interval": "0 10 0 * * ?"
}
}
}{
"result": "updated"
}Comma-separated list of data stream names used to limit the request.
Wildcard (*) expressions are supported. If omitted, all data streams are returned.
Type of data stream that wildcard patterns can match.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
If true, returns all relevant default configurations for the index template.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Whether the maximum timestamp for each data stream should be calculated and returned.
GET _data_stream/my-data-stream
resp = client.indices.get_data_stream(
name="my-data-stream",
)const response = await client.indices.getDataStream({
name: "my-data-stream",
});response = client.indices.get_data_stream(
name: "my-data-stream"
)$resp = $client->indices()->getDataStream([
"name" => "my-data-stream",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/my-data-stream"client.indices().getDataStream(g -> g
.name("my-data-stream")
);
{
"data_streams": [
{
"name": "my-data-stream",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-2099.03.07-000001",
"index_uuid": "xCEhwsp8Tey0-FLNFYVwSg",
"prefer_ilm": true,
"ilm_policy": "my-lifecycle-policy",
"managed_by": "Index Lifecycle Management"
},
{
"index_name": ".ds-my-data-stream-2099.03.08-000002",
"index_uuid": "PA_JquKGSiKcAKBA8DJ5gw",
"prefer_ilm": true,
"ilm_policy": "my-lifecycle-policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 2,
"_meta": {
"my-meta-field": "foo"
},
"status": "GREEN",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": true,
"template": "my-index-template",
"ilm_policy": "my-lifecycle-policy",
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"rollover_on_write": false
},
{
"name": "my-data-stream-two",
"timestamp_field": {
"name": "@timestamp"
},
"indices": [
{
"index_name": ".ds-my-data-stream-two-2099.03.08-000001",
"index_uuid": "3liBu2SYS5axasRt6fUIpA",
"prefer_ilm": true,
"ilm_policy": "my-lifecycle-policy",
"managed_by": "Index Lifecycle Management"
}
],
"generation": 1,
"_meta": {
"my-meta-field": "foo"
},
"status": "YELLOW",
"next_generation_managed_by": "Index Lifecycle Management",
"prefer_ilm": true,
"template": "my-index-template",
"ilm_policy": "my-lifecycle-policy",
"hidden": false,
"system": false,
"allow_custom_routing": false,
"replicated": false,
"rollover_on_write": false
}
]
}Get information about an index or data stream's current data stream lifecycle status, such as time since index creation, time since rollover, the lifecycle configuration managing the index, or any errors encountered during lifecycle execution.
GET .ds-metrics-2023.03.22-000001/_lifecycle/explain
resp = client.indices.explain_data_lifecycle(
index=".ds-metrics-2023.03.22-000001",
)const response = await client.indices.explainDataLifecycle({
index: ".ds-metrics-2023.03.22-000001",
});response = client.indices.explain_data_lifecycle(
index: ".ds-metrics-2023.03.22-000001"
)$resp = $client->indices()->explainDataLifecycle([
"index" => ".ds-metrics-2023.03.22-000001",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/.ds-metrics-2023.03.22-000001/_lifecycle/explain"client.indices().explainDataLifecycle(e -> e
.index(".ds-metrics-2023.03.22-000001")
);
{
"indices": {
".ds-metrics-2023.03.22-000001": {
"index" : ".ds-metrics-2023.03.22-000001",
"managed_by_lifecycle" : true,
"index_creation_date_millis" : 1679475563571,
"time_since_index_creation" : "843ms",
"rollover_date_millis" : 1679475564293,
"time_since_rollover" : "121ms",
"lifecycle" : { },
"generation_time" : "121ms"
}
}{
"indices": {
".ds-metrics-2023.03.22-000001": {
"index" : ".ds-metrics-2023.03.22-000001",
"managed_by_lifecycle" : true,
"index_creation_date_millis" : 1679475563571,
"time_since_index_creation" : "843ms",
"lifecycle" : {
"enabled": true
},
"error": "{\"type\":\"validation_exception\",\"reason\":\"Validation Failed: 1: this action would add [2] shards, but this cluster
currently has [4]/[3] maximum normal shards open;\"}"
}
}Update the data stream lifecycle of the specified data streams.
Comma-separated list of data streams used to limit the request.
Supports wildcards (*).
To target all data streams use * or _all.
Type of data stream that wildcard patterns can match.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and
d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.
If defined, it turns data stream lifecycle on/off (true/false) for this data stream. A data stream lifecycle
that's disabled (enabled: false) will have no effect on the data stream.
Default value is true.
PUT _data_stream/my-data-stream/_lifecycle
{
"data_retention": "7d"
}resp = client.indices.put_data_lifecycle(
name="my-data-stream",
data_retention="7d",
)const response = await client.indices.putDataLifecycle({
name: "my-data-stream",
data_retention: "7d",
});response = client.indices.put_data_lifecycle(
name: "my-data-stream",
body: {
"data_retention": "7d"
}
)$resp = $client->indices()->putDataLifecycle([
"name" => "my-data-stream",
"body" => [
"data_retention" => "7d",
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"data_retention":"7d"}' "$ELASTICSEARCH_URL/_data_stream/my-data-stream/_lifecycle"client.indices().putDataLifecycle(p -> p
.dataRetention(d -> d
.time("7d")
)
.name("my-data-stream")
);
{
"data_retention": "7d"
}{
"downsampling": [
{
"after": "1d",
"fixed_interval": "10m"
},
{
"after": "7d",
"fixed_interval": "1d"
}
]
}{
"acknowledged": true
}A comma-separated list of data streams or data stream patterns. Supports wildcards (*).
GET /_data_stream/my-data-stream/_mappings
resp = client.indices.get_data_stream_mappings(
name="my-data-stream",
)const response = await client.indices.getDataStreamMappings({
name: "my-data-stream",
});response = client.indices.get_data_stream_mappings(
name: "my-data-stream"
)$resp = $client->indices()->getDataStreamMappings([
"name" => "my-data-stream",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/my-data-stream/_mappings"{
"data_streams": [
{
"name": "my-data-stream",
"mappings": {
"properties": {
"field1": {
"type": "ip"
},
"field3": {
"type": "text"
}
}
},
"effective_mappings": {
"properties": {
"field1": {
"type": "ip"
},
"field2": {
"type": "text"
},
"field3": {
"type": "text"
}
}
}
}
]
}Get the data stream options configuration of one or more data streams.
Comma-separated list of data streams to limit the request.
Supports wildcards (*).
To target all data streams, omit this parameter or use * or _all.
Type of data stream that wildcard patterns can match.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
curl \
--request GET 'http://api.example.com/_data_stream/{name}/_options' \
--header "Authorization: $API_KEY"Update the data stream options of the specified data streams.
Comma-separated list of data streams used to limit the request.
Supports wildcards (*).
To target all data streams use * or _all.
Type of data stream that wildcard patterns can match.
Supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
curl \
--request PUT 'http://api.example.com/_data_stream/{name}/_options' \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '{"failure_store":{"enabled":true,"lifecycle":{"data_retention":"string","enabled":true}}}'A comma-separated list of data streams or data stream patterns. Supports wildcards (*).
GET /_data_stream/my-data-stream/_settings
resp = client.indices.get_data_stream_settings(
name="my-data-stream",
)const response = await client.indices.getDataStreamSettings({
name: "my-data-stream",
});response = client.indices.get_data_stream_settings(
name: "my-data-stream"
)$resp = $client->indices()->getDataStreamSettings([
"name" => "my-data-stream",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/_data_stream/my-data-stream/_settings"{
"data_streams": [
{
"name": "my-data-stream",
"settings": {
"index": {
"lifecycle": {
"name": "new-test-policy"
},
"number_of_shards": "11"
}
},
"effective_settings": {
"index": {
"lifecycle": {
"name": "new-test-policy"
},
"mode": "standard",
"number_of_shards": "11",
"number_of_replicas": "0"
}
}
}
]
}This API can be used to override settings on specific data streams. These overrides will take precedence over what is specified in the template that the data stream matches. To prevent your data stream from getting into an invalid state, only certain settings are allowed. If possible, the setting change is applied to all backing indices. Otherwise, it will be applied when the data stream is next rolled over.
manageIf true, the request does not actually change the settings on any data streams or indices. Instead, it
simulates changing the settings and reports back to the user what would have happened had these settings
actually been applied.
The period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
The period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are -1 or 0.
PUT /_data_stream/my-data-stream/_settings
{
"index.lifecycle.name" : "new-test-policy",
"index.number_of_shards": 11
}resp = client.indices.put_data_stream_settings(
name="my-data-stream",
settings={
"index.lifecycle.name": "new-test-policy",
"index.number_of_shards": 11
},
)const response = await client.indices.putDataStreamSettings({
name: "my-data-stream",
settings: {
"index.lifecycle.name": "new-test-policy",
"index.number_of_shards": 11,
},
});response = client.indices.put_data_stream_settings(
name: "my-data-stream",
body: {
"index.lifecycle.name": "new-test-policy",
"index.number_of_shards": 11
}
)$resp = $client->indices()->putDataStreamSettings([
"name" => "my-data-stream",
"body" => [
"index.lifecycle.name" => "new-test-policy",
"index.number_of_shards" => 11,
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"index.lifecycle.name":"new-test-policy","index.number_of_shards":11}' "$ELASTICSEARCH_URL/_data_stream/my-data-stream/_settings"{
"index.lifecycle.name" : "new-test-policy",
"index.number_of_shards": 11
}{
"data_streams": [
{
"name": "my-data-stream",
"applied_to_data_stream": true,
"settings": {
"index": {
"lifecycle": {
"name": "new-test-policy"
},
"number_of_shards": "11"
}
},
"effective_settings": {
"index": {
"lifecycle": {
"name": "new-test-policy"
},
"mode": "standard",
"number_of_shards": "11",
"number_of_replicas": "0"
}
},
"index_settings_results": {
"applied_to_data_stream_only": [
"index.number_of_shards"
],
"applied_to_data_stream_and_backing_indices": [
"index.lifecycle.name"
]
}
}
]
}{
"data_streams": [
{
"name": "my-data-stream",
"applied_to_data_stream": true,
"settings": {
"index": {
"lifecycle": {
"name": "new-test-policy"
},
"number_of_shards": "11"
}
},
"effective_settings": {
"index": {
"lifecycle": {
"name": "new-test-policy"
},
"mode": "standard",
"number_of_shards": "11",
"number_of_replicas": "0"
}
},
"index_settings_results": {
"applied_to_data_stream_only": [
"index.number_of_shards"
],
"applied_to_data_stream_and_backing_indices": [
"index.lifecycle.name"
],
"errors": [
{
"index": ".ds-my-data-stream-2025.05.28-000001",
"error": "index [.ds-my-data-stream-2025.05.28-000001] blocked by: [FORBIDDEN/9/index metadata (api)];"
}
]
}
}
]
}{
"data_streams": [
{
"name": "my-data-stream",
"applied_to_data_stream": false,
"error": "Cannot set the following settings on a data stream: [index.number_of_replicas]",
"settings": {},
"effective_settings": {},
"index_settings_results": {
"applied_to_data_stream_only": [],
"applied_to_data_stream_and_backing_indices": []
}
}
]
}Performs one or more data stream modification actions in a single atomic operation.
POST _data_stream/_modify
{
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}resp = client.indices.modify_data_stream(
actions=[
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
],
)const response = await client.indices.modifyDataStream({
actions: [
{
remove_backing_index: {
data_stream: "my-data-stream",
index: ".ds-my-data-stream-2023.07.26-000001",
},
},
{
add_backing_index: {
data_stream: "my-data-stream",
index: ".ds-my-data-stream-2023.07.26-000001-downsample",
},
},
],
});response = client.indices.modify_data_stream(
body: {
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}
)$resp = $client->indices()->modifyDataStream([
"body" => [
"actions" => array(
[
"remove_backing_index" => [
"data_stream" => "my-data-stream",
"index" => ".ds-my-data-stream-2023.07.26-000001",
],
],
[
"add_backing_index" => [
"data_stream" => "my-data-stream",
"index" => ".ds-my-data-stream-2023.07.26-000001-downsample",
],
],
),
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"actions":[{"remove_backing_index":{"data_stream":"my-data-stream","index":".ds-my-data-stream-2023.07.26-000001"}},{"add_backing_index":{"data_stream":"my-data-stream","index":".ds-my-data-stream-2023.07.26-000001-downsample"}}]}' "$ELASTICSEARCH_URL/_data_stream/_modify"client.indices().modifyDataStream(m -> m
.actions(List.of(Action.of(a -> a
.removeBackingIndex(r -> r
.dataStream("my-data-stream")
.index(".ds-my-data-stream-2023.07.26-000001")
)),Action.of(ac -> ac
.addBackingIndex(ad -> ad
.dataStream("my-data-stream")
.index(".ds-my-data-stream-2023.07.26-000001-downsample")
))))
);
{
"actions": [
{
"remove_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001"
}
},
{
"add_backing_index": {
"data_stream": "my-data-stream",
"index": ".ds-my-data-stream-2023.07.26-000001-downsample"
}
}
]
}All methods and paths for this operation:
Add a JSON document to the specified data stream or index and make it searchable. If the target is an index and the document already exists, the request updates the document and increments its version.
NOTE: You cannot use this API to send update requests for existing documents in a data stream.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or index alias:
PUT /<target>/_doc/<_id> request format, you must have the create, index, or write index privilege.POST /<target>/_doc/ request format, you must have the create_doc, create, index, or write index privilege.auto_configure, create_index, or manage index privilege.Automatic data stream creation requires a matching index template with data stream enabled.
NOTE: Replica shards might not all be started when an indexing operation returns successfully.
By default, only the primary is required. Set wait_for_active_shards to change this default behavior.
Automatically create data streams and indices
If the request's target doesn't exist and matches an index template with a data_stream definition, the index operation automatically creates the data stream.
If the target doesn't exist and doesn't match a data stream template, the operation automatically creates the index and applies any matching index templates.
NOTE: Elasticsearch includes several built-in index templates. To avoid naming collisions with these templates, refer to index pattern documentation.
If no mapping exists, the index operation creates a dynamic mapping. By default, new fields and objects are automatically added to the mapping if needed.
Automatic index creation is controlled by the action.auto_create_index setting.
If it is true, any index can be created automatically.
You can modify this setting to explicitly allow or block automatic creation of indices that match specified patterns or set it to false to turn off automatic index creation entirely.
Specify a comma-separated list of patterns you want to allow or prefix each pattern with + or - to indicate whether it should be allowed or blocked.
When a list is specified, the default behaviour is to disallow.
NOTE: The action.auto_create_index setting affects the automatic creation of indices only.
It does not affect the creation of data streams.
Optimistic concurrency control
Index operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters.
If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409.
Routing
By default, shard placement — or routing — is controlled by using a hash of the document's ID value.
For more explicit control, the value fed into the hash function used by the router can be directly specified on a per-operation basis using the routing parameter.
When setting up explicit mapping, you can also use the _routing field to direct the index operation to extract the routing value from the document itself.
This does come at the (very minimal) cost of an additional document parsing pass.
If the _routing mapping is defined and set to be required, the index operation will fail if no routing value is provided or extracted.
NOTE: Data streams do not support custom routing unless they were created with the allow_custom_routing setting enabled in the template.
Distributed
The index operation is directed to the primary shard based on its route and performed on the actual node containing this shard. After the primary shard completes the operation, if needed, the update is distributed to applicable replicas.
Active shards
To improve the resiliency of writes to the system, indexing operations can be configured to wait for a certain number of active shard copies before proceeding with the operation.
If the requisite number of active shard copies are not available, then the write operation must wait and retry, until either the requisite shard copies have started or a timeout occurs.
By default, write operations only wait for the primary shards to be active before proceeding (that is to say wait_for_active_shards is 1).
This default can be overridden in the index settings dynamically by setting index.write.wait_for_active_shards.
To alter this behavior per operation, use the wait_for_active_shards request parameter.
Valid values are all or any positive integer up to the total number of configured copies per shard in the index (which is number_of_replicas+1).
Specifying a negative value or a number greater than the number of shard copies will throw an error.
For example, suppose you have a cluster of three nodes, A, B, and C and you create an index index with the number of replicas set to 3 (resulting in 4 shard copies, one more copy than there are nodes).
If you attempt an indexing operation, by default the operation will only ensure the primary copy of each shard is available before proceeding.
This means that even if B and C went down and A hosted the primary shard copies, the indexing operation would still proceed with only one copy of the data.
If wait_for_active_shards is set on the request to 3 (and all three nodes are up), the indexing operation will require 3 active shard copies before proceeding.
This requirement should be met because there are 3 active nodes in the cluster, each one holding a copy of the shard.
However, if you set wait_for_active_shards to all (or to 4, which is the same in this situation), the indexing operation will not proceed as you do not have all 4 copies of each shard active in the index.
The operation will timeout unless a new node is brought up in the cluster to host the fourth copy of the shard.
It is important to note that this setting greatly reduces the chances of the write operation not writing to the requisite number of shard copies, but it does not completely eliminate the possibility, because this check occurs before the write operation starts.
After the write operation is underway, it is still possible for replication to fail on any number of shard copies but still succeed on the primary.
The _shards section of the API response reveals the number of shard copies on which replication succeeded and failed.
No operation (noop) updates
When updating a document by using this API, a new version of the document is always created even if the document hasn't changed.
If this isn't acceptable use the _update API with detect_noop set to true.
The detect_noop option isn't available on this API because it doesn’t fetch the old source and isn't able to compare it against the new source.
There isn't a definitive rule for when noop updates aren't acceptable. It's a combination of lots of factors like how frequently your data source sends updates that are actually noops and how many queries per second Elasticsearch runs on the shard receiving the updates.
Versioning
Each indexed document is given a version number.
By default, internal versioning is used that starts at 1 and increments with each update, deletes included.
Optionally, the version number can be set to an external value (for example, if maintained in a database).
To enable this functionality, version_type should be set to external.
The value provided must be a numeric, long value greater than or equal to 0, and less than around 9.2e+18.
NOTE: Versioning is completely real time, and is not affected by the near real time aspects of search operations. If no version is provided, the operation runs without any version checks.
When using the external version type, the system checks to see if the version number passed to the index request is greater than the version of the currently stored document. If true, the document will be indexed and the new version number used. If the value provided is less than or equal to the stored document's version number, a version conflict will occur and the index operation will fail. For example:
PUT my-index-000001/_doc/1?version=2&version_type=external
{
"user": {
"id": "elkbee"
}
}
In this example, the operation will succeed since the supplied version of 2 is higher than the current document version of 1.
If the document was already updated and its version was set to 2 or higher, the indexing command will fail and result in a conflict (409 HTTP status code).
A nice side effect is that there is no need to maintain strict ordering of async indexing operations run as a result of changes to a source database, as long as version numbers from the source database are used.
Even the simple case of updating the Elasticsearch index using data from a database is simplified if external versioning is used, as only the latest version will be used if the index operations arrive out of order.
## Required authorization
* Index privileges: `index`
The name of the data stream or index to target.
If the target doesn't exist and matches the name or wildcard (*) pattern of an index template with a data_stream definition, this request creates the data stream.
If the target doesn't exist and doesn't match a data stream template, this request creates the index.
You can check for existing targets with the resolve index API.
A unique identifier for the document.
To automatically generate a document ID, use the POST /<target>/_doc/ request format and omit this parameter.
Only perform the operation if the document has this primary term.
Only perform the operation if the document has this sequence number.
True or false if to include the document source in the error message in case of parsing errors.
Set to create to only index the document if it does not already exist (put if absent).
If a document with the specified _id already exists, the indexing operation will fail.
The behavior is the same as using the <index>/_create endpoint.
If a document ID is specified, this paramater defaults to index.
Otherwise, it defaults to create.
If the request targets a data stream, an op_type of create is required.
Supported values include:
index: Overwrite any documents that already exist.create: Only index documents that do not already exist.Values are index or create.
The ID of the pipeline to use to preprocess incoming documents.
If the index has a default ingest pipeline specified, then setting the value to _none disables the default ingest pipeline for this request.
If a final pipeline is configured it will always run, regardless of the value of this parameter.
If true, Elasticsearch refreshes the affected shards to make this operation visible to search.
If wait_for, it waits for a refresh to make this operation visible to search.
If false, it does nothing with refreshes.
Values are true, false, or wait_for.
A custom value that is used to route operations to a specific shard.
The period the request waits for the following operations: automatic index creation, dynamic mapping updates, waiting for active shards.
This parameter is useful for situations where the primary shard assigned to perform the operation might not be available when the operation runs. Some reasons for this might be that the primary shard is currently recovering from a gateway or undergoing relocation. By default, the operation will wait on the primary shard to become available for at least 1 minute before failing and responding with an error. The actual wait time could be longer, particularly when multiple waits occur.
Values are -1 or 0.
An explicit version number for concurrency control. It must be a non-negative long number.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
The number of shard copies that must be active before proceeding with the operation.
You can set it to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
The default value of 1 means it waits for each primary shard to be active.
Values are all or index-setting.
If true, the destination must be an index alias.
If true, the request's actions must target a data stream (existing or to be created).
POST my-index-000001/_doc/
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1 200 1070000",
"user": {
"id": "kimchy"
}
}resp = client.index(
index="my-index-000001",
document={
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1 200 1070000",
"user": {
"id": "kimchy"
}
},
)const response = await client.index({
index: "my-index-000001",
document: {
"@timestamp": "2099-11-15T13:12:00",
message: "GET /search HTTP/1.1 200 1070000",
user: {
id: "kimchy",
},
},
});response = client.index(
index: "my-index-000001",
body: {
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1 200 1070000",
"user": {
"id": "kimchy"
}
}
)$resp = $client->index([
"index" => "my-index-000001",
"body" => [
"@timestamp" => "2099-11-15T13:12:00",
"message" => "GET /search HTTP/1.1 200 1070000",
"user" => [
"id" => "kimchy",
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"@timestamp":"2099-11-15T13:12:00","message":"GET /search HTTP/1.1 200 1070000","user":{"id":"kimchy"}}' "$ELASTICSEARCH_URL/my-index-000001/_doc/"client.index(i -> i
.index("my-index-000001")
.document(JsonData.fromJson("{\"@timestamp\":\"2099-11-15T13:12:00\",\"message\":\"GET /search HTTP/1.1 200 1070000\",\"user\":{\"id\":\"kimchy\"}}"))
);
{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1 200 1070000",
"user": {
"id": "kimchy"
}
}{
"@timestamp": "2099-11-15T13:12:00",
"message": "GET /search HTTP/1.1 200 1070000",
"user": {
"id": "kimchy"
}
}{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
},
"_index": "my-index-000001",
"_id": "W0tpsmIBdwcYyG50zbta",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"result": "created"
}{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
},
"_index": "my-index-000001",
"_id": "1",
"_version": 1,
"_seq_no": 0,
"_primary_term": 1,
"result": "created"
}Remove a JSON document from the specified index.
NOTE: You cannot send deletion requests directly to a data stream. To delete a document in a data stream, you must target the backing index containing the document.
Optimistic concurrency control
Delete operations can be made conditional and only be performed if the last modification to the document was assigned the sequence number and primary term specified by the if_seq_no and if_primary_term parameters.
If a mismatch is detected, the operation will result in a VersionConflictException and a status code of 409.
Versioning
Each document indexed is versioned.
When deleting a document, the version can be specified to make sure the relevant document you are trying to delete is actually being deleted and it has not changed in the meantime.
Every write operation run on a document, deletes included, causes its version to be incremented.
The version number of a deleted document remains available for a short time after deletion to allow for control of concurrent operations.
The length of time for which a deleted document's version remains available is determined by the index.gc_deletes index setting.
Routing
If routing is used during indexing, the routing value also needs to be specified to delete a document.
If the _routing mapping is set to required and no routing value is specified, the delete API throws a RoutingMissingException and rejects the request.
For example:
DELETE /my-index-000001/_doc/1?routing=shard-1
This request deletes the document with ID 1, but it is routed based on the user. The document is not deleted if the correct routing is not specified.
Distributed
The delete operation gets hashed into a specific shard ID. It then gets redirected into the primary shard within that ID group and replicated (if needed) to shard replicas within that ID group.
deleteOnly perform the operation if the document has this primary term.
Only perform the operation if the document has this sequence number.
If true, Elasticsearch refreshes the affected shards to make this operation visible to search.
If wait_for, it waits for a refresh to make this operation visible to search.
If false, it does nothing with refreshes.
Values are true, false, or wait_for.
A custom value used to route operations to a specific shard.
The period to wait for active shards.
This parameter is useful for situations where the primary shard assigned to perform the delete operation might not be available when the delete operation runs. Some reasons for this might be that the primary shard is currently recovering from a store or undergoing relocation. By default, the delete operation will wait on the primary shard to become available for up to 1 minute before failing and responding with an error.
Values are -1 or 0.
An explicit version number for concurrency control. It must match the current version of the document for the request to succeed.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
The minimum number of shard copies that must be active before proceeding with the operation.
You can set it to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
The default value of 1 means it waits for each primary shard to be active.
Values are all or index-setting.
DELETE /my-index-000001/_doc/1
resp = client.delete(
index="my-index-000001",
id="1",
)const response = await client.delete({
index: "my-index-000001",
id: 1,
});response = client.delete(
index: "my-index-000001",
id: "1"
)$resp = $client->delete([
"index" => "my-index-000001",
"id" => "1",
]);curl -X DELETE -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_doc/1"client.delete(d -> d
.id("1")
.index("my-index-000001")
);
{
"_shards": {
"total": 2,
"failed": 0,
"successful": 2
},
"_index": "my-index-000001",
"_id": "1",
"_version": 2,
"_primary_term": 1,
"_seq_no": 5,
"result": "deleted"
}Deletes documents that match the specified query.
If the Elasticsearch security features are enabled, you must have the following index privileges for the target data stream, index, or alias:
readdelete or writeYou can specify the query criteria in the request URI or the request body using the same syntax as the search API. When you submit a delete by query request, Elasticsearch gets a snapshot of the data stream or index when it begins processing the request and deletes matching documents using internal versioning. If a document changes between the time that the snapshot is taken and the delete operation is processed, it results in a version conflict and the delete operation fails.
NOTE: Documents with a version equal to 0 cannot be deleted using delete by query because internal versioning does not support 0 as a valid version number.
While processing a delete by query request, Elasticsearch performs multiple search requests sequentially to find all of the matching documents to delete. A bulk delete request is performed for each batch of matching documents. If a search or bulk request is rejected, the requests are retried up to 10 times, with exponential back off. If the maximum retry limit is reached, processing halts and all failed requests are returned in the response. Any delete requests that completed successfully still stick, they are not rolled back.
You can opt to count version conflicts instead of halting and returning by setting conflicts to proceed.
Note that if you opt to count version conflicts the operation could attempt to delete more documents from the source than max_docs until it has successfully deleted max_docs documents, or it has gone through every document in the source query.
Throttling delete requests
To control the rate at which delete by query issues batches of delete operations, you can set requests_per_second to any positive decimal number.
This pads each batch with a wait time to throttle the rate.
Set requests_per_second to -1 to disable throttling.
Throttling uses a wait time between batches so that the internal scroll requests can be given a timeout that takes the request padding into account.
The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing.
By default the batch size is 1000, so if requests_per_second is set to 500:
target_time = 1000 / 500 per second = 2 seconds
wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
Since the batch is issued as a single _bulk request, large batch sizes cause Elasticsearch to create many requests and wait before starting the next set.
This is "bursty" instead of "smooth".
Slicing
Delete by query supports sliced scroll to parallelize the delete process. This can improve efficiency and provide a convenient way to break the request down into smaller parts.
Setting slices to auto lets Elasticsearch choose the number of slices to use.
This setting will use one slice per shard, up to a certain limit.
If there are multiple source data streams or indices, it will choose the number of slices based on the index or backing index with the smallest number of shards.
Adding slices to the delete by query operation creates sub-requests which means it has some quirks:
slices will rethrottle the unfinished sub-request proportionally.slices will cancel each sub-request.slices each sub-request won't get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.requests_per_second and max_docs on a request with slices are distributed proportionally to each sub-request. Combine that with the earlier point about distribution being uneven and you should conclude that using max_docs with slices might not result in exactly max_docs documents being deleted.If you're slicing manually or otherwise tuning automatic slicing, keep in mind that:
slices hurts performance. Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.Whether query or delete performance dominates the runtime depends on the documents being reindexed and cluster resources.
Cancel a delete by query operation
Any delete by query can be canceled using the task cancel API. For example:
POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel
The task ID can be found by using the get tasks API.
Cancellation should happen quickly but might take a few seconds. The get task status API will continue to list the delete by query task until this task checks that it has been cancelled and terminates itself.
read,deleteA comma-separated list of data streams, indices, and aliases to search.
It supports wildcards (*).
To search all data streams or indices, omit this parameter or use * or _all.
If false, the request returns an error if any wildcard expression, index alias, or _all value targets only missing or closed indices.
This behavior applies even if the request targets other open indices.
For example, a request targeting foo*,bar* returns an error if an index starts with foo but no index starts with bar.
Analyzer to use for the query string.
This parameter can be used only when the q query string parameter is specified.
If true, wildcard and prefix queries are analyzed.
This parameter can be used only when the q query string parameter is specified.
What to do if delete by query hits version conflicts: abort or proceed.
Supported values include:
abort: Stop reindexing if there are conflicts.proceed: Continue reindexing even if there are conflicts.Values are abort or proceed.
The default operator for query string query: AND or OR.
This parameter can be used only when the q query string parameter is specified.
Values are and, AND, or, or OR.
The field to use as default where no field prefix is given in the query string.
This parameter can be used only when the q query string parameter is specified.
The type of index that wildcard patterns can match.
If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams.
It supports comma-separated values, such as open,hidden.
Supported values include:
all: Match any data stream or index, including hidden ones.open: Match open, non-hidden indices. Also matches any non-hidden data stream.closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.none: Wildcard expressions are not accepted.Values are all, open, closed, hidden, or none.
Skips the specified number of documents.
If true, format-based query failures (such as providing text to a numeric field) in the query string will be ignored.
This parameter can be used only when the q query string parameter is specified.
The maximum number of documents to process.
Defaults to all documents.
When set to a value less then or equal to scroll_size, a scroll will not be used to retrieve the results for the operation.
The node or shard the operation should be performed on. It is random by default.
If true, Elasticsearch refreshes all shards involved in the delete by query after the request completes.
This is different than the delete API's refresh parameter, which causes just the shard that received the delete request to be refreshed.
Unlike the delete API, it does not support wait_for.
If true, the request cache is used for this request.
Defaults to the index-level setting.
The throttle for this request in sub-requests per second.
A custom value used to route operations to a specific shard.
A query in the Lucene query string syntax.
The period to retain the search context for scrolling.
Values are -1 or 0.
The size of the scroll request that powers the operation.
The explicit timeout for each search request. It defaults to no timeout.
Values are -1 or 0.
The type of the search operation.
Available options include query_then_fetch and dfs_query_then_fetch.
Supported values include:
query_then_fetch: Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.dfs_query_then_fetch: Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.Values are query_then_fetch or dfs_query_then_fetch.
The number of slices this task should be divided into.
Value is auto.
A comma-separated list of <field>:<direction> pairs.
The specific tag of the request for logging and statistical purposes.
The maximum number of documents to collect for each shard. If a query reaches this limit, Elasticsearch terminates the query early. Elasticsearch collects documents before sorting.
Use with caution. Elasticsearch applies this parameter to each shard handling the request. When possible, let Elasticsearch perform early termination automatically. Avoid specifying this parameter for requests that target data streams with backing indices across multiple data tiers.
The period each deletion request waits for active shards.
Values are -1 or 0.
If true, returns the document version as part of a hit.
The number of shard copies that must be active before proceeding with the operation.
Set to all or any positive integer up to the total number of shards in the index (number_of_replicas+1).
The timeout value controls how long each write request waits for unavailable shards to become available.
Values are all or index-setting.
If true, the request blocks until the operation is complete.
If false, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task. Elasticsearch creates a record of this task as a document at .tasks/task/${taskId}. When you are done with a task, you should delete the task document so Elasticsearch can reclaim the space.
The maximum number of documents to delete.
An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.
POST /my-index-000001,my-index-000002/_delete_by_query
{
"query": {
"match_all": {}
}
}resp = client.delete_by_query(
index="my-index-000001,my-index-000002",
query={
"match_all": {}
},
)const response = await client.deleteByQuery({
index: "my-index-000001,my-index-000002",
query: {
match_all: {},
},
});response = client.delete_by_query(
index: "my-index-000001,my-index-000002",
body: {
"query": {
"match_all": {}
}
}
)$resp = $client->deleteByQuery([
"index" => "my-index-000001,my-index-000002",
"body" => [
"query" => [
"match_all" => new ArrayObject([]),
],
],
]);curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"query":{"match_all":{}}}' "$ELASTICSEARCH_URL/my-index-000001,my-index-000002/_delete_by_query"client.deleteByQuery(d -> d
.index(List.of("my-index-000001","my-index-000002"))
.query(q -> q
.matchAll(m -> m)
)
);
{
"query": {
"match_all": {}
}
}{
"query": {
"term": {
"user.id": "kimchy"
}
},
"max_docs": 1
}{
"slice": {
"id": 0,
"max": 2
},
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}{
"query": {
"range": {
"http.response.bytes": {
"lt": 2000000
}
}
}
}{
"took" : 147,
"timed_out": false,
"total": 119,
"deleted": 119,
"batches": 1,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1.0,
"throttled_until_millis": 0,
"failures" : [ ]
}Get the source of a document. For example:
GET my-index-000001/_source/1
You can use the source filtering parameters to control which parts of the _source are returned:
GET my-index-000001/_source/1/?_source_includes=*.id&_source_excludes=entities
readThe node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.
If true, the request is real-time as opposed to near-real-time.
If true, the request refreshes the relevant shards before retrieving the document.
Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
A custom value used to route operations to a specific shard.
Indicates whether to return the _source field (true or false) or lists the fields to return.
A comma-separated list of source fields to exclude in the response.
A comma-separated list of source fields to include in the response.
The version number for concurrency control. It must match the current version of the document for the request to succeed.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
GET my-index-000001/_source/1
resp = client.get_source(
index="my-index-000001",
id="1",
)const response = await client.getSource({
index: "my-index-000001",
id: 1,
});response = client.get_source(
index: "my-index-000001",
id: "1"
)$resp = $client->getSource([
"index" => "my-index-000001",
"id" => "1",
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_source/1"client.getSource(g -> g
.id("1")
.index("my-index-000001")
);
Check whether a document source exists in an index. For example:
HEAD my-index-000001/_source/1
A document's source is not available if it is disabled in the mapping.
readA comma-separated list of data streams, indices, and aliases.
It supports wildcards (*).
A unique identifier for the document.
The node or shard the operation should be performed on. By default, the operation is randomized between the shard replicas.
If true, the request is real-time as opposed to near-real-time.
If true, the request refreshes the relevant shards before retrieving the document.
Setting it to true should be done after careful thought and verification that this does not cause a heavy load on the system (and slow down indexing).
A custom value used to route operations to a specific shard.
Indicates whether to return the _source field (true or false) or lists the fields to return.
A comma-separated list of source fields to exclude in the response.
A comma-separated list of source fields to include in the response.
The version number for concurrency control. It must match the current version of the document for the request to succeed.
The version type.
Supported values include:
internal: Use internal versioning that starts at 1 and increments with each update or delete.external: Only index the document if the specified version is strictly higher than the version of the stored document or if there is no existing document.external_gte: Only index the document if the specified version is equal or higher than the version of the stored document or if there is no existing document.
NOTE: The external_gte version type is meant for special use cases and should be used with care.
If used incorrectly, it can result in loss of data.force: This option is deprecated because it can cause primary and replica shards to diverge.Values are internal, external, external_gte, or force.
HEAD my-index-000001/_source/1
resp = client.exists_source(
index="my-index-000001",
id="1",
)const response = await client.existsSource({
index: "my-index-000001",
id: 1,
});response = client.exists_source(
index: "my-index-000001",
id: "1"
)$resp = $client->existsSource([
"index" => "my-index-000001",
"id" => "1",
]);curl --head -H "Authorization: ApiKey $ELASTIC_API_KEY" "$ELASTICSEARCH_URL/my-index-000001/_source/1"client.existsSource(e -> e
.id("1")
.index("my-index-000001")
);
All methods and paths for this operation:
Get multiple JSON documents by ID from one or more indices. If you specify an index in the request URI, you only need to specify the document IDs in the request body. To ensure fast responses, this multi get (mget) API responds with partial results if one or more shards fail.
Filter source fields
By default, the _source field is returned for every document (if stored).
Use the _source and _source_include or source_exclude attributes to filter what fields are returned for a particular document.
You can include the _source, _source_includes, and _source_excludes query parameters in the request URI to specify the defaults to use when there are no per-document instructions.
Get stored fields
Use the stored_fields attribute to specify the set of stored fields you want to retrieve.
Any requested fields that are not stored are ignored.
You can include the stored_fields query parameter in the request URI to specify the defaults to use when there are no per-document instructions.
readName of the index to retrieve documents from when ids are specified, or when a document in the docs array does not specify an index.
Specifies the node or shard the operation should be performed on. Random by default.
If true, the request is real-time as opposed to near-real-time.
If true, the request refreshes relevant shards before retrieving documents.
Custom value used to route operations to a specific shard.
True or false to return the _source field or not, or a list of fields to return.
A comma-separated list of source fields to exclude from the response.
You can also use this parameter to exclude fields from the subset specified in _source_includes query parameter.
A comma-separated list of source fields to include in the response.
If this parameter is specified, only these source fields are returned. You can exclude fields from this subset using the _source_excludes query parameter.
If the _source parameter is false, this parameter is ignored.
If true, retrieves the document fields stored in the index rather than the document _source.
GET /my-index-000001/_mget
{
"docs": [
{
"_id": "1"
},
{
"_id": "2"
}
]
}resp = client.mget(
index="my-index-000001",
docs=[
{
"_id": "1"
},
{
"_id": "2"
}
],
)const response = await client.mget({
index: "my-index-000001",
docs: [
{
_id: "1",
},
{
_id: "2",
},
],
});response = client.mget(
index: "my-index-000001",
body: {
"docs": [
{
"_id": "1"
},
{
"_id": "2"
}
]
}
)$resp = $client->mget([
"index" => "my-index-000001",
"body" => [
"docs" => array(
[
"_id" => "1",
],
[
"_id" => "2",
],
),
],
]);curl -X GET -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"docs":[{"_id":"1"},{"_id":"2"}]}' "$ELASTICSEARCH_URL/my-index-000001/_mget"client.mget(m -> m
.docs(List.of(MultiGetOperation.of(mu -> mu
.id("1")),MultiGetOperation.of(mu -> mu
.id("2"))))
.index("my-index-000001")
);
{
"docs": [
{
"_id": "1"
},
{
"_id": "2"
}
]
}{
"docs": [
{
"_index": "test",
"_id": "1",
"_source": false
},
{
"_index": "test",
"_id": "2",
"_source": [ "field3", "field4" ]
},
{
"_index": "test",
"_id": "3",
"_source": {
"include": [ "user" ],
"exclude": [ "user.location" ]
}
}
]
}{
"docs": [
{
"_index": "test",
"_id": "1",
"stored_fields": [ "field1", "field2" ]
},
{
"_index": "test",
"_id": "2",
"stored_fields": [ "field3", "field4" ]
}
]
}{
"docs": [
{
"_index": "test",
"_id": "1",
"routing": "key2"
},
{
"_index": "test",
"_id": "2"
}
]
}