This method implements equation (3) in [1]. In this equation the probability
of the decided label being correct is used to estimate the calibration
property of the predictor.
References
[1]: Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger,
On Calibration of Modern Neural Networks.
Proceedings of the 34th International Conference on Machine Learning
(ICML 2017).
arXiv:1706.04599
https://arxiv.org/pdf/1706.04599.pdf
Args
num_bins
int, number of probability bins, e.g. 10.
logits
Tensor, (n,nlabels), with logits for n instances and nlabels.
labels_true
Tensor, (n,), with tf.int32 or tf.int64 elements containing
ground truth class labels in the range [0,nlabels].
labels_predicted
Tensor, (n,), with tf.int32 or tf.int64 elements
containing decisions of the predictive system. If None, we will use
the argmax decision using the logits.
name
Python str name prefixed to Ops created by this function.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2023-11-21 UTC."],[],[],null,["# tfp.stats.expected_calibration_error\n\n\u003cbr /\u003e\n\n|----------------------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/probability/blob/v0.23.0/tensorflow_probability/python/stats/calibration.py#L258-L319) |\n\nCompute the Expected Calibration Error (ECE). \n\n tfp.stats.expected_calibration_error(\n num_bins, logits=None, labels_true=None, labels_predicted=None, name=None\n )\n\nThis method implements equation (3) in \\[1\\]. In this equation the probability\nof the decided label being correct is used to estimate the calibration\nproperty of the predictor.\n| **Note:** a trade-off exist between using a small number of `num_bins` and the estimation reliability of the ECE. In particular, this method may produce unreliable ECE estimates in case there are few samples available in some bins. As an alternative to this method, consider also using `bayesian_expected_calibration_error`.\n\n#### References\n\n\\[1\\]: Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger,\nOn Calibration of Modern Neural Networks.\nProceedings of the 34th International Conference on Machine Learning\n(ICML 2017).\narXiv:1706.04599\n\u003chttps://arxiv.org/pdf/1706.04599.pdf\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| `num_bins` | int, number of probability bins, e.g. 10. |\n| `logits` | Tensor, (n,nlabels), with logits for n instances and nlabels. |\n| `labels_true` | Tensor, (n,), with tf.int32 or tf.int64 elements containing ground truth class labels in the range \\[0,nlabels\\]. |\n| `labels_predicted` | Tensor, (n,), with tf.int32 or tf.int64 elements containing decisions of the predictive system. If `None`, we will use the argmax decision using the `logits`. |\n| `name` | Python `str` name prefixed to Ops created by this function. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ------- ||\n|-------|-----------------------------|\n| `ece` | Tensor, scalar, tf.float32. |\n\n\u003cbr /\u003e"]]
View source on GitHub