Path parameters
-
The identifier for the anomaly detection job. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.
Query parameters
-
If
true
, wildcard indices expressions that resolve into no concrete indices are ignored. This includes the_all
string or when no indices are specified. -
Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values.
Supported values include:
all
: Match any data stream or index, including hidden ones.open
: Match open, non-hidden indices. Also matches any non-hidden data stream.closed
: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.hidden
: Match hidden data streams and hidden indices. Must be combined withopen
,closed
, orboth
.none
: Wildcard expressions are not accepted.
Values are
all
,open
,closed
,hidden
, ornone
. -
If
true
, concrete, expanded or aliased indices are ignored when frozen.
Body
Required
-
Advanced configuration option. Specifies whether this job can open when there is insufficient machine learning node capacity for it to be immediately assigned to a node. By default, if a machine learning node with capacity to run the job cannot immediately be found, the open anomaly detection jobs API returns an error. However, this is also subject to the cluster-wide
xpack.ml.max_lazy_ml_nodes
setting. If this option is set to true, the open anomaly detection jobs API does not return an error and the job waits in the opening state until sufficient machine learning node capacity is available.Default value is
false
. -
Specifies how to analyze the data. After you create a job, you cannot change the analysis configuration; all the properties are informational.
-
Limits can be applied for the resources required to hold the mathematical models in memory. These limits are approximate and can be set per job. They do not control the memory used by other processes, for example the Elasticsearch Java processes.
-
Advanced configuration option. The time between each periodic persistence of the model. The default value is a randomized value between 3 to 4 hours, which avoids all jobs persisting at exactly the same time. The smallest allowed value is 1 hour. For very large models (several GB), persistence could take 10-20 minutes, so do not set the
background_persist_interval
value too low. -
Advanced configuration option. Contains custom meta data about the job.
-
Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies a period of time (in days) after which only the first snapshot per day is retained. This period is relative to the timestamp of the most recent snapshot for this job. Valid values range from 0 to
model_snapshot_retention_days
.Default value is
1
. -
Defines the format of the input data when you send data to the job by using the post data API. Note that when configure a datafeed, these properties are automatically set. When data is received via the post data API, it is not stored in Elasticsearch. Only the results for anomaly detection are retained.
-
Defines a datafeed for the anomaly detection job. If Elasticsearch security features are enabled, your datafeed remembers which roles the user who created it had at the time of creation and runs the query using those same roles. If you provide secondary authorization headers, those credentials are used instead.
-
A description of the job.
-
The identifier for the anomaly detection job. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.
-
A list of job groups. A job can belong to no groups or many.
-
This advanced configuration option stores model information along with the results. It provides a more detailed view into anomaly detection. If you enable model plot it can add considerable overhead to the performance of the system; it is not feasible for jobs with many entities. Model plot provides a simplified and indicative view of the model and its bounds. It does not display complex features such as multivariate correlations or multimodal data. As such, anomalies may occasionally be reported which cannot be seen in the model plot. Model plot config can be configured when the job is created or updated later. It must be disabled if performance issues are experienced.
-
Advanced configuration option, which affects the automatic removal of old model snapshots for this job. It specifies the maximum period of time (in days) that snapshots are retained. This period is relative to the timestamp of the most recent snapshot for this job. By default, snapshots ten days older than the newest snapshot are deleted.
Default value is
10
. -
Advanced configuration option. The period over which adjustments to the score are applied, as new data is seen. The default value is the longer of 30 days or 100 bucket spans.
-
A text string that affects the name of the machine learning results index. By default, the job generates an index named
.ml-anomalies-shared
. -
Advanced configuration option. The period of time (in days) that results are retained. Age is calculated relative to the timestamp of the latest bucket result. If this property has a non-null value, once per day at 00:30 (server time), results that are the specified number of days older than the latest bucket result are deleted from Elasticsearch. The default value is null, which means all results are retained. Annotations generated by the system also count as results for retention purposes; they are deleted after the same number of days as results. Annotations added by users are retained forever.
PUT /_ml/anomaly_detectors/job-01
{
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "Sum of bytes",
"function": "sum",
"field_name": "bytes"
}
]
},
"data_description": {
"time_field": "timestamp",
"time_format": "epoch_ms"
},
"analysis_limits": {
"model_memory_limit": "11MB"
},
"model_plot_config": {
"enabled": true,
"annotations_enabled": true
},
"results_index_name": "test-job1",
"datafeed_config": {
"indices": [
"kibana_sample_data_logs"
],
"query": {
"bool": {
"must": [
{
"match_all": {}
}
]
}
},
"runtime_mappings": {
"hour_of_day": {
"type": "long",
"script": {
"source": "emit(doc['timestamp'].value.getHour());"
}
}
},
"datafeed_id": "datafeed-test-job1"
}
}
resp = client.ml.put_job(
job_id="job-01",
analysis_config={
"bucket_span": "15m",
"detectors": [
{
"detector_description": "Sum of bytes",
"function": "sum",
"field_name": "bytes"
}
]
},
data_description={
"time_field": "timestamp",
"time_format": "epoch_ms"
},
analysis_limits={
"model_memory_limit": "11MB"
},
model_plot_config={
"enabled": True,
"annotations_enabled": True
},
results_index_name="test-job1",
datafeed_config={
"indices": [
"kibana_sample_data_logs"
],
"query": {
"bool": {
"must": [
{
"match_all": {}
}
]
}
},
"runtime_mappings": {
"hour_of_day": {
"type": "long",
"script": {
"source": "emit(doc['timestamp'].value.getHour());"
}
}
},
"datafeed_id": "datafeed-test-job1"
},
)
const response = await client.ml.putJob({
job_id: "job-01",
analysis_config: {
bucket_span: "15m",
detectors: [
{
detector_description: "Sum of bytes",
function: "sum",
field_name: "bytes",
},
],
},
data_description: {
time_field: "timestamp",
time_format: "epoch_ms",
},
analysis_limits: {
model_memory_limit: "11MB",
},
model_plot_config: {
enabled: true,
annotations_enabled: true,
},
results_index_name: "test-job1",
datafeed_config: {
indices: ["kibana_sample_data_logs"],
query: {
bool: {
must: [
{
match_all: {},
},
],
},
},
runtime_mappings: {
hour_of_day: {
type: "long",
script: {
source: "emit(doc['timestamp'].value.getHour());",
},
},
},
datafeed_id: "datafeed-test-job1",
},
});
response = client.ml.put_job(
job_id: "job-01",
body: {
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "Sum of bytes",
"function": "sum",
"field_name": "bytes"
}
]
},
"data_description": {
"time_field": "timestamp",
"time_format": "epoch_ms"
},
"analysis_limits": {
"model_memory_limit": "11MB"
},
"model_plot_config": {
"enabled": true,
"annotations_enabled": true
},
"results_index_name": "test-job1",
"datafeed_config": {
"indices": [
"kibana_sample_data_logs"
],
"query": {
"bool": {
"must": [
{
"match_all": {}
}
]
}
},
"runtime_mappings": {
"hour_of_day": {
"type": "long",
"script": {
"source": "emit(doc['timestamp'].value.getHour());"
}
}
},
"datafeed_id": "datafeed-test-job1"
}
}
)
$resp = $client->ml()->putJob([
"job_id" => "job-01",
"body" => [
"analysis_config" => [
"bucket_span" => "15m",
"detectors" => array(
[
"detector_description" => "Sum of bytes",
"function" => "sum",
"field_name" => "bytes",
],
),
],
"data_description" => [
"time_field" => "timestamp",
"time_format" => "epoch_ms",
],
"analysis_limits" => [
"model_memory_limit" => "11MB",
],
"model_plot_config" => [
"enabled" => true,
"annotations_enabled" => true,
],
"results_index_name" => "test-job1",
"datafeed_config" => [
"indices" => array(
"kibana_sample_data_logs",
),
"query" => [
"bool" => [
"must" => array(
[
"match_all" => new ArrayObject([]),
],
),
],
],
"runtime_mappings" => [
"hour_of_day" => [
"type" => "long",
"script" => [
"source" => "emit(doc['timestamp'].value.getHour());",
],
],
],
"datafeed_id" => "datafeed-test-job1",
],
],
]);
curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"analysis_config":{"bucket_span":"15m","detectors":[{"detector_description":"Sum of bytes","function":"sum","field_name":"bytes"}]},"data_description":{"time_field":"timestamp","time_format":"epoch_ms"},"analysis_limits":{"model_memory_limit":"11MB"},"model_plot_config":{"enabled":true,"annotations_enabled":true},"results_index_name":"test-job1","datafeed_config":{"indices":["kibana_sample_data_logs"],"query":{"bool":{"must":[{"match_all":{}}]}},"runtime_mappings":{"hour_of_day":{"type":"long","script":{"source":"emit(doc['"'"'timestamp'"'"'].value.getHour());"}}},"datafeed_id":"datafeed-test-job1"}}' "$ELASTICSEARCH_URL/_ml/anomaly_detectors/job-01"
client.ml().putJob(p -> p
.analysisConfig(a -> a
.bucketSpan(b -> b
.time("15m")
)
.detectors(d -> d
.detectorDescription("Sum of bytes")
.fieldName("bytes")
.function("sum")
)
)
.analysisLimits(an -> an
.modelMemoryLimit("11MB")
)
.dataDescription(d -> d
.timeField("timestamp")
.timeFormat("epoch_ms")
)
.datafeedConfig(d -> d
.datafeedId("datafeed-test-job1")
.indices("kibana_sample_data_logs")
.query(q -> q
.bool(b -> b
.must(m -> m
.matchAll(ma -> ma)
)
)
)
.runtimeMappings("hour_of_day", r -> r
.script(s -> s
.source(so -> so
.scriptString("emit(doc['timestamp'].value.getHour());")
)
)
.type(RuntimeFieldType.Long)
)
)
.jobId("job-01")
.modelPlotConfig(m -> m
.annotationsEnabled(true)
.enabled(true)
)
.resultsIndexName("test-job1")
);
{
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "Sum of bytes",
"function": "sum",
"field_name": "bytes"
}
]
},
"data_description": {
"time_field": "timestamp",
"time_format": "epoch_ms"
},
"analysis_limits": {
"model_memory_limit": "11MB"
},
"model_plot_config": {
"enabled": true,
"annotations_enabled": true
},
"results_index_name": "test-job1",
"datafeed_config": {
"indices": [
"kibana_sample_data_logs"
],
"query": {
"bool": {
"must": [
{
"match_all": {}
}
]
}
},
"runtime_mappings": {
"hour_of_day": {
"type": "long",
"script": {
"source": "emit(doc['timestamp'].value.getHour());"
}
}
},
"datafeed_id": "datafeed-test-job1"
}
}
{
"job_id": "test-job1",
"job_type": "anomaly_detector",
"job_version": "8.4.0",
"create_time": 1656087283340,
"datafeed_config": {
"datafeed_id": "datafeed-test-job1",
"job_id": "test-job1",
"authorization": {
"roles": [
"superuser"
]
},
"query_delay": "61499ms",
"chunking_config": {
"mode": "auto"
},
"indices_options": {
"expand_wildcards": [
"open"
],
"ignore_unavailable": false,
"allow_no_indices": true,
"ignore_throttled": true
},
"query": {
"bool": {
"must": [
{
"match_all": {}
}
]
}
},
"indices": [
"kibana_sample_data_logs"
],
"scroll_size": 1000,
"delayed_data_check_config": {
"enabled": true
},
"runtime_mappings": {
"hour_of_day": {
"type": "long",
"script": {
"source": "emit(doc['timestamp'].value.getHour());"
}
}
}
},
"analysis_config": {
"bucket_span": "15m",
"detectors": [
{
"detector_description": "Sum of bytes",
"function": "sum",
"field_name": "bytes",
"detector_index": 0
}
],
"influencers": [],
"model_prune_window": "30d"
},
"analysis_limits": {
"model_memory_limit": "11mb",
"categorization_examples_limit": 4
},
"data_description": {
"time_field": "timestamp",
"time_format": "epoch_ms"
},
"model_plot_config": {
"enabled": true,
"annotations_enabled": true
},
"model_snapshot_retention_days": 10,
"daily_model_snapshot_retention_after_days": 1,
"results_index_name": "custom-test-job1",
"allow_lazy_open": false
}