Manage time series data without data streams

Stack

Even though data streams are a convenient way to scale and manage time series data, they are designed to be append-only. We recognise there might be use-cases where data needs to be updated or deleted in place and the data streams don’t support delete and update requests directly, so the index APIs would need to be used directly on the data stream’s backing indices. In these cases we still recommend using a data stream.

If you frequently send multiple documents using the same _id expecting last-write-wins, you can use an index alias instead of a data stream to manage indices containing the time series data and periodically roll over to a new index.

To automate rollover and management of time series indices with ILM using an index alias, you:

1. Create a lifecycle policy that defines the appropriate phases and actions.
2. Create an index template to apply the policy to each new index.
3. Bootstrap an index as the initial write index.
4. Verify indices are moving through the lifecycle phases as expected.

A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to five phases: hot, warm, cold, frozen, and delete.

For example, you might define a policy named timeseries_policy that has the following two phases:

• A hot phase that defines a rollover action to specify that an index rolls over when it reaches either a max_primary_shard_size of 50 gigabytes or a max_age of 30 days.
• A delete phase that sets min_age to remove the index 90 days after rollover.

You can create the policy in Kibana or with the create or update policy API.

Kibana

To create the policy from Kibana, open the menu and go to Stack Management > Index Lifecycle Policies. Click Create policy.

×

API

Use the Create or update policy API to add an ILM policy to the Elasticsearch cluster:

				PUT _ilm/policy/timeseries_policy
					{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
		

1. The min_age defaults to 0ms, so new indices enter the hot phase immediately.
2. Trigger the rollover action when either of the conditions are met.
3. Move the index into the delete phase 90 days after rollover.
4. Trigger the delete action when the index enters the delete phase.

To automatically apply a lifecycle policy to the new write index on rollover, specify the policy in the index template used to create new indices.

For example, you might create a timeseries_template that is applied to new indices whose names match the timeseries-* index pattern.

To enable automatic rollover, the template configures two ILM settings:

• index.lifecycle.name specifies the name of the lifecycle policy to apply to new indices that match the index pattern.
• index.lifecycle.rollover_alias specifies the index alias to be rolled over when the rollover action is triggered for an index.

				PUT _index_template/timeseries_template
					{
  "index_patterns": ["timeseries-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "timeseries_policy",
      "index.lifecycle.rollover_alias": "timeseries"
    }
  }
}
		

To get things started, you need to bootstrap an initial index and designate it as the write index for the rollover alias specified in your index template. The name of this index must match the template’s index pattern and end with a number. On rollover, this value is incremented to generate a name for the new index.

For example, the following request creates an index called timeseries-000001 and makes it the write index for the timeseries alias.

				PUT timeseries-000001
					{
  "aliases": {
    "timeseries": {
      "is_write_index": true
    }
  }
}
		

When the rollover conditions are met, the rollover action:

• Creates a new index called timeseries-000002. This matches the timeseries-* pattern, so the settings from timeseries_template are applied to the new index.
• Designates the new index as the write index and makes the bootstrap index read-only.

This process repeats each time rollover conditions are met. You can search across all of the indices managed by the timeseries_policy with the timeseries alias. Write operations should be sent towards the alias, which will route them to its current write index.

Retrieving the status information for managed indices is very similar to the case of managing time series data with data streams case, the only difference being the indices namespace. Run the following API request to retrieve the lifecycle progress:

				GET timeseries-*/_ilm/explain
		

See Check lifecycle progress for more information.