tfp.substrates.numpy.stats.brier_decomposition
Stay organized with collections
Save and categorize content based on your preferences.
Decompose the Brier score into uncertainty, resolution, and reliability.
tfp.substrates.numpy.stats.brier_decomposition(
labels, logits, name=None
)
[Proper scoring rules][1] measure the quality of probabilistic predictions;
any proper scoring rule admits a [unique decomposition][2] as
Score = Uncertainty - Resolution + Reliability, where:
Uncertainty, is a generalized entropy of the average predictive
distribution; it can both be positive or negative.Resolution, is a generalized variance of individual predictive
distributions; it is always non-negative. Difference in predictions reveal
information, that is why a larger resolution improves the predictive score.Reliability, a measure of calibration of predictions against the true
frequency of events. It is always non-negative and a lower value here
indicates better calibration.
This method estimates the above decomposition for the case of the Brier
scoring rule for discrete outcomes. For this, we need to discretize the space
of probability distributions; we choose a simple partition of the space into
nlabels events: given a distribution p over nlabels outcomes, the index
k for which p_k > p_i for all i != k determines the discretization
outcome; that is, p in M_k, where M_k is the set of all distributions for
which p_k is the largest value among all probabilities.
The estimation error of each component is O(k/n), where n is the number
of instances and k is the number of labels. There may be an error of this
order when compared to brier_score.
References
[1]: Tilmann Gneiting, Adrian E. Raftery.
Strictly Proper Scoring Rules, Prediction, and Estimation.
Journal of the American Statistical Association, Vol. 102, 2007.
https://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf
[2]: Jochen Broecker. Reliability, sufficiency, and the decomposition of
proper scores.
Quarterly Journal of the Royal Meteorological Society, Vol. 135, 2009.
https://rmets.onlinelibrary.wiley.com/doi/epdf/10.1002/qj.456
Args |
|---|
labels
|
Tensor, (n,), with tf.int32 or tf.int64 elements containing ground
truth class labels in the range [0,nlabels-1].
|
logits
|
Tensor, (n, nlabels), with logits for n instances and nlabels.
|
name
|
Python str name prefixed to Ops created by this function.
|
Returns |
|---|
uncertainty
|
Tensor, scalar, the uncertainty component of the
decomposition.
|
resolution
|
Tensor, scalar, the resolution component of the decomposition.
|
reliability
|
Tensor, scalar, the reliability component of the
decomposition.
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2023-11-21 UTC.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2023-11-21 UTC."],[],[],null,["# tfp.substrates.numpy.stats.brier_decomposition\n\n\u003cbr /\u003e\n\n|-------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/probability/blob/v0.23.0/tensorflow_probability/substrates/numpy/stats/calibration.py#L40-L156) |\n\nDecompose the Brier score into uncertainty, resolution, and reliability.\n\n#### View aliases\n\n\n**Main aliases**\n\n[`tfp.experimental.substrates.numpy.stats.brier_decomposition`](https://www.tensorflow.org/probability/api_docs/python/tfp/substrates/numpy/stats/brier_decomposition)\n\n\u003cbr /\u003e\n\n tfp.substrates.numpy.stats.brier_decomposition(\n labels, logits, name=None\n )\n\n\\[Proper scoring rules\\]\\[1\\] measure the quality of probabilistic predictions;\nany proper scoring rule admits a \\[unique decomposition\\]\\[2\\] as\n`Score = Uncertainty - Resolution + Reliability`, where:\n\n- `Uncertainty`, is a generalized entropy of the average predictive distribution; it can both be positive or negative.\n- `Resolution`, is a generalized variance of individual predictive distributions; it is always non-negative. Difference in predictions reveal information, that is why a larger resolution improves the predictive score.\n- `Reliability`, a measure of calibration of predictions against the true frequency of events. It is always non-negative and a lower value here indicates better calibration.\n\nThis method estimates the above decomposition for the case of the Brier\nscoring rule for discrete outcomes. For this, we need to discretize the space\nof probability distributions; we choose a simple partition of the space into\n`nlabels` events: given a distribution `p` over `nlabels` outcomes, the index\n`k` for which `p_k \u003e p_i` for all `i != k` determines the discretization\noutcome; that is, `p in M_k`, where `M_k` is the set of all distributions for\nwhich `p_k` is the largest value among all probabilities.\n\nThe estimation error of each component is O(k/n), where n is the number\nof instances and k is the number of labels. There may be an error of this\norder when compared to `brier_score`.\n\n#### References\n\n\\[1\\]: Tilmann Gneiting, Adrian E. Raftery.\nStrictly Proper Scoring Rules, Prediction, and Estimation.\nJournal of the American Statistical Association, Vol. 102, 2007.\n\u003chttps://www.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf\u003e\n\\[2\\]: Jochen Broecker. Reliability, sufficiency, and the decomposition of\nproper scores.\nQuarterly Journal of the Royal Meteorological Society, Vol. 135, 2009.\n\u003chttps://rmets.onlinelibrary.wiley.com/doi/epdf/10.1002/qj.456\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|----------|---------------------------------------------------------------------------------------------------------------------|\n| `labels` | Tensor, (n,), with tf.int32 or tf.int64 elements containing ground truth class labels in the range \\[0,nlabels-1\\]. |\n| `logits` | Tensor, (n, nlabels), with logits for n instances and nlabels. |\n| `name` | Python `str` name prefixed to Ops created by this function. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ------- ||\n|---------------|-----------------------------------------------------------------|\n| `uncertainty` | Tensor, scalar, the uncertainty component of the decomposition. |\n| `resolution` | Tensor, scalar, the resolution component of the decomposition. |\n| `reliability` | Tensor, scalar, the reliability component of the decomposition. |\n\n\u003cbr /\u003e"]]
View source on GitHub