Manage time series data with data streams

Stack

To automate rollover and management of a data stream with ILM, you:

1. Create a lifecycle policy that defines the appropriate phases and actions.
2. Create an index template to create the data stream and apply the ILM policy and the indices settings and mappings configurations for the backing indices.
3. Verify that indices are moving through the lifecycle phases as expected.

A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to five phases: hot, warm, cold, frozen, and delete.

For example, you might define a policy named timeseries_policy that has the following two phases:

• A hot phase that defines a rollover action to specify that an index rolls over when it reaches either a max_primary_shard_size of 50 gigabytes or a max_age of 30 days.
• A delete phase that sets min_age to remove the index 90 days after rollover.

You can create the policy in Kibana or with the create or update policy API.

Kibana

To create the policy from Kibana, open the menu and go to Stack Management > Index Lifecycle Policies. Click Create policy.

×

API

Use the Create or update policy API to add an ILM policy to the Elasticsearch cluster:

				PUT _ilm/policy/timeseries_policy
					{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
		

1. The min_age defaults to 0ms, so new indices enter the hot phase immediately.
2. Trigger the rollover action when either of the conditions are met.
3. Move the index into the delete phase 90 days after rollover.
4. Trigger the delete action when the index enters the delete phase.

To set up a data stream, first create an index template to specify the lifecycle policy. Because the template is for a data stream, it must also include a data_stream definition.

For example, you might create a template named timeseries_template and use it for a future data stream named timeseries.

To enable ILM to manage the data stream, the template configures one ILM setting:

• index.lifecycle.name specifies the name of the lifecycle policy that you want to apply to the data stream.

Use the Kibana Create template wizard to add a template or the Create or update index template API to add a template and apply the lifecycle policy to indices matching the template.

				PUT _index_template/timeseries_template
					{
  "index_patterns": ["timeseries"],
  "data_stream": { },
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "timeseries_policy"
    }
  }
}
		

To get things started, index a document into the name or wildcard pattern defined in the index_patterns of the index template. As long as an existing data stream, index, or index alias does not already use the name, the index request automatically creates a corresponding data stream with a single backing index. Elasticsearch automatically indexes the request’s documents into this backing index, which also acts as the stream’s write index.

For example, the following request creates the timeseries data stream and the first generation backing index called .ds-timeseries-2099.03.08-000001.

				POST timeseries/_doc
					{
  "message": "logged the request",
  "@timestamp": "1591890611"
}
		

When a rollover condition in the lifecycle policy is met, the rollover action:

• Creates the second generation backing index, named .ds-timeseries-2099.03.08-000002. Because it is a backing index of the timeseries data stream, the configuration from the timeseries_template index template is applied to the new index.
• As it is the latest generation index of the timeseries data stream, the newly created backing index .ds-timeseries-2099.03.08-000002 becomes the data stream’s write index.

This process repeats each time a rollover condition is met. You can search across all of the data stream’s backing indices, managed by the timeseries_policy, with the timeseries data stream name. Write operations should be sent to the data stream name, which will route them to its current write index. Read operations against the data stream will be handled by all its backing indices.

Use Kibana to view the current status of your managed indices and details about the ILM policy, or the ILM explain API. Find out things like:

• What phase an index is in and when it entered that phase.
• The current action and what step is being performed.
• If any errors have occurred or progress is blocked.

For example, the following request gets information about the timeseries data stream’s backing indices:

				GET .ds-timeseries-*/_ilm/explain
		

The following response shows the data stream’s first generation backing index is waiting for the hot phase’s rollover action. It remains in this state and ILM continues to call check-rollover-ready until a rollover condition is met.

{
  "indices": {
    ".ds-timeseries-2099.03.07-000001": {
      "index": ".ds-timeseries-2099.03.07-000001",
      "index_creation_date_millis": 1538475653281,
      "time_since_index_creation": "30s",
      "managed": true,
      "policy": "timeseries_policy",
      "lifecycle_date_millis": 1538475653281,
      "age": "30s",
      "phase": "hot",
      "phase_time_millis": 1538475653317,
      "action": "rollover",
      "action_time_millis": 1538475653317,
      "step": "check-rollover-ready",
      "step_time_millis": 1538475653317,
      "phase_execution": {
        "policy": "timeseries_policy",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_primary_shard_size": "50gb",
              "max_age": "30d"
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1539609701576
      }
    }
  }
}
		

1. The age of the index used for calculating when to rollover the index via the max_age
2. The policy used to manage the index
3. The age of the indexed used to transition to the next phase (in this case it is the same with the age of the index).
4. The step ILM is performing on the index
5. The definition of the current phase (the hot phase)