Evaluate data frame analytics Generally available

POST /_ml/data_frame/_evaluate
Api key auth

The API packages together commonly used evaluation metrics for various types of machine learning features. This has been designed for use on indexes created by data frame analytics. Evaluation requires both a ground truth field and an analytics result field to be present.

Required authorization

  • Cluster privileges: monitor_ml
application/json

Body Required

  • evaluation object Required
    Hide evaluation attributes Show evaluation attributes object
    • classification object
      Hide classification attributes Show classification attributes object
      • actual_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • predicted_field string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • top_classes_field string

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • metrics object
        Hide metrics attributes Show metrics attributes object
        • auc_roc object
          Hide auc_roc attributes Show auc_roc attributes object
          • class_name string
          • include_curve boolean

            Whether or not the curve should be returned in addition to the score. Default value is false.

        • precision object

          Precision of predictions (per-class and average).

          Hide precision attribute Show precision attribute object
          • * object Additional properties
        • recall object

          Recall of predictions (per-class and average).

          Hide recall attribute Show recall attribute object
          • * object Additional properties
        • accuracy object

          Accuracy of predictions (per-class and overall).

          Hide accuracy attribute Show accuracy attribute object
          • * object Additional properties
        • multiclass_confusion_matrix object

          Multiclass confusion matrix.

          Hide multiclass_confusion_matrix attribute Show multiclass_confusion_matrix attribute object
          • * object Additional properties
    • outlier_detection object
      Hide outlier_detection attributes Show outlier_detection attributes object
      • actual_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • predicted_probability_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • metrics object
        Hide metrics attributes Show metrics attributes object
        • auc_roc object
          Hide auc_roc attributes Show auc_roc attributes object
          • class_name string
          • include_curve boolean

            Whether or not the curve should be returned in addition to the score. Default value is false.

        • precision object

          Precision of predictions (per-class and average).

          Hide precision attribute Show precision attribute object
          • * object Additional properties
        • recall object

          Recall of predictions (per-class and average).

          Hide recall attribute Show recall attribute object
          • * object Additional properties
        • confusion_matrix object

          Accuracy of predictions (per-class and overall).

          Hide confusion_matrix attribute Show confusion_matrix attribute object
          • * object Additional properties
    • regression object
      Hide regression attributes Show regression attributes object
      • actual_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • predicted_field string Required

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • metrics object
        Hide metrics attributes Show metrics attributes object
        • mse object

          Average squared difference between the predicted values and the actual (ground truth) value. For more information, read this wiki article.

          Hide mse attribute Show mse attribute object
          • * object Additional properties
        • msle object
          Hide msle attribute Show msle attribute object
          • offset number

            Defines the transition point at which you switch from minimizing quadratic error to minimizing quadratic log error. Defaults to 1.

        • huber object
          Hide huber attribute Show huber attribute object
          • delta number

            Approximates 1/2 (prediction - actual)2 for values much less than delta and approximates a straight line with slope delta for values much larger than delta. Defaults to 1. Delta needs to be greater than 0.

        • r_squared object

          Proportion of the variance in the dependent variable that is predictable from the independent variables.

          Hide r_squared attribute Show r_squared attribute object
          • * object Additional properties
  • index string Required
  • query object

    An Elasticsearch Query DSL (Domain Specific Language) object that defines a query.

    External documentation

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • classification object
      Hide classification attributes Show classification attributes object
      • auc_roc object
        Hide auc_roc attributes Show auc_roc attributes object
        • value number Required
        • curve array[object]
          Hide curve attributes Show curve attributes object
          • tpr number Required
          • fpr number Required
          • threshold number Required
      • accuracy object
        Hide accuracy attributes Show accuracy attributes object
        • classes array[object] Required
          Hide classes attributes Show classes attributes object
          • value number Required
          • class_name string Required
        • overall_accuracy number Required
      • multiclass_confusion_matrix object
        Hide multiclass_confusion_matrix attributes Show multiclass_confusion_matrix attributes object
        • confusion_matrix array[object] Required
          Hide confusion_matrix attributes Show confusion_matrix attributes object
          • actual_class string Required
          • actual_class_doc_count number Required
          • predicted_classes array[object] Required
          • other_predicted_class_doc_count number Required
        • other_actual_class_count number Required
      • precision object
        Hide precision attributes Show precision attributes object
        • classes array[object] Required
          Hide classes attributes Show classes attributes object
          • value number Required
          • class_name string Required
        • avg_precision number Required
      • recall object
        Hide recall attributes Show recall attributes object
        • classes array[object] Required
          Hide classes attributes Show classes attributes object
          • value number Required
          • class_name string Required
        • avg_recall number Required
    • outlier_detection object
      Hide outlier_detection attributes Show outlier_detection attributes object
      • auc_roc object
        Hide auc_roc attributes Show auc_roc attributes object
        • value number Required
        • curve array[object]
          Hide curve attributes Show curve attributes object
          • tpr number Required
          • fpr number Required
          • threshold number Required
      • precision object

        Set the different thresholds of the outlier score at where the metric is calculated.

        Hide precision attribute Show precision attribute object
        • * number Additional properties
      • recall object

        Set the different thresholds of the outlier score at where the metric is calculated.

        Hide recall attribute Show recall attribute object
        • * number Additional properties
      • confusion_matrix object

        Set the different thresholds of the outlier score at where the metrics (tp - true positive, fp - false positive, tn - true negative, fn - false negative) are calculated.

        Hide confusion_matrix attribute Show confusion_matrix attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • tp number Required

            True Positive

          • fp number Required

            False Positive

          • tn number Required

            True Negative

          • fn number Required

            False Negative

    • regression object
      Hide regression attributes Show regression attributes object
      • huber object
        Hide huber attribute Show huber attribute object
        • value number Required
      • mse object
        Hide mse attribute Show mse attribute object
        • value number Required
      • msle object
        Hide msle attribute Show msle attribute object
        • value number Required
      • r_squared object
        Hide r_squared attribute Show r_squared attribute object
        • value number Required
POST /_ml/data_frame/_evaluate 
POST _ml/data_frame/_evaluate
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "predicted_field": "ml.animal_class_prediction",
      "metrics": {
        "multiclass_confusion_matrix": {}
      }
    }
  }
}
resp = client.ml.evaluate_data_frame(
    index="animal_classification",
    evaluation={
        "classification": {
            "actual_field": "animal_class",
            "predicted_field": "ml.animal_class_prediction",
            "metrics": {
                "multiclass_confusion_matrix": {}
            }
        }
    },
)
const response = await client.ml.evaluateDataFrame({
  index: "animal_classification",
  evaluation: {
    classification: {
      actual_field: "animal_class",
      predicted_field: "ml.animal_class_prediction",
      metrics: {
        multiclass_confusion_matrix: {},
      },
    },
  },
});
response = client.ml.evaluate_data_frame(
  body: {
    "index": "animal_classification",
    "evaluation": {
      "classification": {
        "actual_field": "animal_class",
        "predicted_field": "ml.animal_class_prediction",
        "metrics": {
          "multiclass_confusion_matrix": {}
        }
      }
    }
  }
)
$resp = $client->ml()->evaluateDataFrame([
    "body" => [
        "index" => "animal_classification",
        "evaluation" => [
            "classification" => [
                "actual_field" => "animal_class",
                "predicted_field" => "ml.animal_class_prediction",
                "metrics" => [
                    "multiclass_confusion_matrix" => new ArrayObject([]),
                ],
            ],
        ],
    ],
]);
curl -X POST -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"index":"animal_classification","evaluation":{"classification":{"actual_field":"animal_class","predicted_field":"ml.animal_class_prediction","metrics":{"multiclass_confusion_matrix":{}}}}}' "$ELASTICSEARCH_URL/_ml/data_frame/_evaluate"
client.ml().evaluateDataFrame(e -> e
    .evaluation(ev -> ev
        .classification(c -> c
            .actualField("animal_class")
            .predictedField("ml.animal_class_prediction")
            .metrics(m -> m)
        )
    )
    .index("animal_classification")
);
Request examples
Run `POST _ml/data_frame/_evaluate` to evaluate a a classification job for an annotated index. The `actual_field` contains the ground truth for classification. The `predicted_field` contains the predicted value calculated by the classification analysis. 
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "predicted_field": "ml.animal_class_prediction",
      "metrics": {
        "multiclass_confusion_matrix": {}
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate a classification job with AUC ROC metrics for an annotated index. The `actual_field` contains the ground truth value for the actual animal classification. This is required in order to evaluate results. The `class_name` specifies the class name that is treated as positive during the evaluation, all the other classes are treated as negative. 
{
  "index": "animal_classification",
  "evaluation": {
    "classification": {
      "actual_field": "animal_class",
      "metrics": {
        "auc_roc": {
          "class_name": "dog"
        }
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate an outlier detection job for an annotated index. 
{
  "index": "my_analytics_dest_index",
  "evaluation": {
    "outlier_detection": {
      "actual_field": "is_outlier",
      "predicted_probability_field": "ml.outlier_score"
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate the testing error of a regression job for an annotated index. The term query in the body limits evaluation to be performed on the test split only. The `actual_field` contains the ground truth for house prices. The `predicted_field` contains the house price calculated by the regression analysis. 
{
  "index": "house_price_predictions",
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "ml.is_training": false
          }
        }
      ]
    }
  },
  "evaluation": {
    "regression": {
      "actual_field": "price",
      "predicted_field": "ml.price_prediction",
      "metrics": {
        "r_squared": {},
        "mse": {},
        "msle": {
          "offset": 10
        },
        "huber": {
          "delta": 1.5
        }
      }
    }
  }
}
Run `POST _ml/data_frame/_evaluate` to evaluate the training error of a regression job for an annotated index. The term query in the body limits evaluation to be performed on the training split only. The `actual_field` contains the ground truth for house prices. The `predicted_field` contains the house price calculated by the regression analysis. 
{
  "index": "house_price_predictions",
  "query": {
    "term": {
      "ml.is_training": {
        "value": true
      }
    }
  },
  "evaluation": {
    "regression": {
      "actual_field": "price",
      "predicted_field": "ml.price_prediction",
      "metrics": {
        "r_squared": {},
        "mse": {},
        "msle": {},
        "huber": {}
      }
    }
  }
}
Response examples (200)
A succesful response from `POST _ml/data_frame/_evaluate` to evaluate a classification analysis job for an annotated index. The `actual_class` contains the name of the class the analysis tried to predict. The `actual_class_doc_count` is the number of documents in the index belonging to the `actual_class`. The `predicted_classes` object contains the list of the predicted classes and the number of predictions associated with the class. 
{
  "classification": {
    "multiclass_confusion_matrix": {
      "confusion_matrix": [
        {
          "actual_class": "cat",
          "actual_class_doc_count": 12,
          "predicted_classes": [
            {
              "predicted_class": "cat",
              "count": 12
            },
            {
              "predicted_class": "dog",
              "count": 0
            }
          ],
          "other_predicted_class_doc_count": 0
        },
        {
          "actual_class": "dog",
          "actual_class_doc_count": 11,
          "predicted_classes": [
            {
              "predicted_class": "dog",
              "count": 7
            },
            {
              "predicted_class": "cat",
              "count": 4
            }
          ],
          "other_predicted_class_doc_count": 0
        }
      ],
      "other_actual_class_count": 0
    }
  }
}
A succesful response from `POST _ml/data_frame/_evaluate` to evaluate a classification analysis job with the AUC ROC metrics for an annotated index. 
{
  "classification": {
    "auc_roc": {
      "value": 0.8941788639536681
    }
  }
}
A successful response from `POST _ml/data_frame/_evaluate` to evaluate an outlier detection job. 
{
  "outlier_detection": {
    "auc_roc": {
      "value": 0.9258475774641445
    },
    "confusion_matrix": {
      "0.25": {
        "tp": 5,
        "fp": 9,
        "tn": 204,
        "fn": 5
      },
      "0.5": {
        "tp": 1,
        "fp": 5,
        "tn": 208,
        "fn": 9
      },
      "0.75": {
        "tp": 0,
        "fp": 4,
        "tn": 209,
        "fn": 10
      }
    },
    "precision": {
      "0.25": 0.35714285714285715,
      "0.5": 0.16666666666666666,
      "0.75": 0
    },
    "recall": {
      "0.25": 0.5,
      "0.5": 0.1,
      "0.75": 0
    }
  }
}