Create a transform
Generally available
Creates a transform.
A transform copies data from source indices, transforms it, and persists it into an entity-centric destination index. You can also think of the destination index as a two-dimensional tabular data structure (known as a data frame). The ID for each document in the data frame is generated from a hash of the entity, so there is a unique row per entity.
You must choose either the latest or pivot method for your transform; you cannot use both in a single transform. If
you choose to use the pivot method for your transform, the entities are defined by the set of group_by fields in
the pivot object. If you choose to use the latest method, the entities are defined by the unique_key field values
in the latest object.
You must have create_index, index, and read privileges on the destination index and read and
view_index_metadata privileges on the source indices. When Elasticsearch security features are enabled, the
transform remembers which roles the user that created it had at the time of creation and uses those same roles. If
those roles do not have the required privileges on the source and destination indices, the transform fails when it
attempts unauthorized operations.
NOTE: You must use Kibana or this API to create a transform. Do not add a transform directly into any
.transform-internal* indices using the Elasticsearch index API. If Elasticsearch security features are enabled, do
not give users any privileges on .transform-internal* indices. If you used transforms prior to 7.5, also do not
give users any privileges on .data-frame-internal* indices.
Required authorization
- Index privileges:
create_index,read,index,view_index_metadata - Cluster privileges:
manage_transform
Path parameters
-
Identifier for the transform. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It has a 64 character limit and must start and end with alphanumeric characters.
Query parameters
-
When the transform is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the source indices and a check that the destination index is not part of the source index pattern. You can use this parameter to skip the checks, for example when the source index does not exist until after the transform is created. The validations are always run when you start the transform, however, with the exception of privilege checks.
-
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Values are
-1or0.
Body
Required
-
The destination for the transform.
-
Free text description of the transform.
-
The interval between checks for changes in the source indices when the transform is running continuously. Also determines the retry interval in the event of transient failures while the transform is searching or indexing. The minimum value is
1sand the maximum is1h. -
The latest method transforms the data by finding the latest document for each unique key.
-
Defines optional transform metadata.
-
The pivot method transforms the data by aggregating and grouping it. These objects define the group by fields and the aggregation to reduce the data.
-
Defines a retention policy for the transform. Data that meets the defined criteria is deleted from the destination index.
-
Defines optional transform settings.
-
The source of the data for the transform.
-
Defines the properties transforms require to run continuously.
PUT _transform/ecommerce_transform1
{
"source": {
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
"pivot": {
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": true
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
"description": "Maximum priced ecommerce data by customer_id in Asia",
"dest": {
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
"frequency": "5m",
"sync": {
"time": {
"field": "order_date",
"delay": "60s"
}
},
"retention_policy": {
"time": {
"field": "order_date",
"max_age": "30d"
}
}
}resp = client.transform.put_transform(
transform_id="ecommerce_transform1",
source={
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
pivot={
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": True
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
description="Maximum priced ecommerce data by customer_id in Asia",
dest={
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
frequency="5m",
sync={
"time": {
"field": "order_date",
"delay": "60s"
}
},
retention_policy={
"time": {
"field": "order_date",
"max_age": "30d"
}
},
)const response = await client.transform.putTransform({
transform_id: "ecommerce_transform1",
source: {
index: "kibana_sample_data_ecommerce",
query: {
term: {
"geoip.continent_name": {
value: "Asia",
},
},
},
},
pivot: {
group_by: {
customer_id: {
terms: {
field: "customer_id",
missing_bucket: true,
},
},
},
aggregations: {
max_price: {
max: {
field: "taxful_total_price",
},
},
},
},
description: "Maximum priced ecommerce data by customer_id in Asia",
dest: {
index: "kibana_sample_data_ecommerce_transform1",
pipeline: "add_timestamp_pipeline",
},
frequency: "5m",
sync: {
time: {
field: "order_date",
delay: "60s",
},
},
retention_policy: {
time: {
field: "order_date",
max_age: "30d",
},
},
});response = client.transform.put_transform(
transform_id: "ecommerce_transform1",
body: {
"source": {
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
"pivot": {
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": true
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
"description": "Maximum priced ecommerce data by customer_id in Asia",
"dest": {
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
"frequency": "5m",
"sync": {
"time": {
"field": "order_date",
"delay": "60s"
}
},
"retention_policy": {
"time": {
"field": "order_date",
"max_age": "30d"
}
}
}
)$resp = $client->transform()->putTransform([
"transform_id" => "ecommerce_transform1",
"body" => [
"source" => [
"index" => "kibana_sample_data_ecommerce",
"query" => [
"term" => [
"geoip.continent_name" => [
"value" => "Asia",
],
],
],
],
"pivot" => [
"group_by" => [
"customer_id" => [
"terms" => [
"field" => "customer_id",
"missing_bucket" => true,
],
],
],
"aggregations" => [
"max_price" => [
"max" => [
"field" => "taxful_total_price",
],
],
],
],
"description" => "Maximum priced ecommerce data by customer_id in Asia",
"dest" => [
"index" => "kibana_sample_data_ecommerce_transform1",
"pipeline" => "add_timestamp_pipeline",
],
"frequency" => "5m",
"sync" => [
"time" => [
"field" => "order_date",
"delay" => "60s",
],
],
"retention_policy" => [
"time" => [
"field" => "order_date",
"max_age" => "30d",
],
],
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"source":{"index":"kibana_sample_data_ecommerce","query":{"term":{"geoip.continent_name":{"value":"Asia"}}}},"pivot":{"group_by":{"customer_id":{"terms":{"field":"customer_id","missing_bucket":true}}},"aggregations":{"max_price":{"max":{"field":"taxful_total_price"}}}},"description":"Maximum priced ecommerce data by customer_id in Asia","dest":{"index":"kibana_sample_data_ecommerce_transform1","pipeline":"add_timestamp_pipeline"},"frequency":"5m","sync":{"time":{"field":"order_date","delay":"60s"}},"retention_policy":{"time":{"field":"order_date","max_age":"30d"}}}' "$ELASTICSEARCH_URL/_transform/ecommerce_transform1"client.transform().putTransform(p -> p
.description("Maximum priced ecommerce data by customer_id in Asia")
.dest(d -> d
.index("kibana_sample_data_ecommerce_transform1")
.pipeline("add_timestamp_pipeline")
)
.frequency(f -> f
.time("5m")
)
.pivot(pi -> pi
.aggregations("max_price", a -> a
.max(m -> m
.field("taxful_total_price")
)
)
.groupBy("customer_id", g -> g
.terms(t -> t
.field("customer_id")
.missingBucket(true)
)
)
)
.retentionPolicy(r -> r
.time(t -> t
.field("order_date")
.maxAge(m -> m
.time("30d")
)
)
)
.source(s -> s
.index("kibana_sample_data_ecommerce")
.query(q -> q
.term(te -> te
.field("geoip.continent_name")
.value(FieldValue.of("Asia"))
)
)
)
.sync(sy -> sy
.time(ti -> ti
.delay(d -> d
.time("60s")
)
.field("order_date")
)
)
.transformId("ecommerce_transform1")
);
{
"source": {
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
"pivot": {
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": true
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
"description": "Maximum priced ecommerce data by customer_id in Asia",
"dest": {
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
"frequency": "5m",
"sync": {
"time": {
"field": "order_date",
"delay": "60s"
}
},
"retention_policy": {
"time": {
"field": "order_date",
"max_age": "30d"
}
}
}{
"source": {
"index": "kibana_sample_data_ecommerce"
},
"latest": {
"unique_key": [
"customer_id"
],
"sort": "order_date"
},
"description": "Latest order for each customer",
"dest": {
"index": "kibana_sample_data_ecommerce_transform2"
},
"frequency": "5m",
"sync": {
"time": {
"field": "order_date",
"delay": "60s"
}
}
}{
"acknowledged": true
}