DOC: incorrect example for stats.cramervonmises

[Yalikesifulei]

Issue with current documentation:

Documentation contains three examples of using cramervonmises, but all of them seem to have discrepancies between values of res.statistic, res.pvalue and descriptions below. For example, first one:

>>> import numpy as np
>>> from scipy import stats
>>> rng = np.random.default_rng()
>>> x = stats.norm.rvs(size=500, random_state=rng)
>>> res = stats.cramervonmises(x, 'norm')
>>> res.statistic, res.pvalue
(0.49121480855028343, 0.04189256516661377)

And the description is "The p-value 0.79 exceeds our chosen significance level, so we do not reject the null hypothesis that the observed sample is drawn from the standard normal distribution". Indeed, the null hypothesis shouldn't be rejected here as x clearly has standard normal distribution.
The same code on my laptop outputs far more different values, which are closer to what is described:

>>> res.statistic, res.pvalue
(0.10486572818633237, 0.561716331166968)

Two other examples share the same issue (see screenshot).

Idea or request for content:

Check and fix the numbers in code examples.

Additional context (e.g. screenshots, GIFs)

[mdhaber]

Thanks @Yalikesifulei. We'll fix this shortly.

[mdhaber]

This example was originally written like:

from scipy import stats
np.random.seed(626)
x = stats.norm.rvs(size=500)
res = stats.cramervonmises(x, 'norm')
res.statistic, res.pvalue  # (0.06342154705518796, 0.792680516270629)

so that is where "The p-value 0.79..." comes from. (See 1.6.0 documentation.)

The reason that the values in the text are no longer consistent with the output of the code is that gh-13863 changed the seed used to generate the data under the hood.

import numpy as np
from scipy import stats
rng = np.random.default_rng(1638083107694713882823079058616272161)
x = stats.norm.rvs(size=500, random_state=rng)
res = stats.cramervonmises(x, 'norm')
res.statistic, res.pvalue  # (0.49121480855028343, 0.04189256516661377)

This is an unlucky seed for this example because it generates data that is unusually extreme, but gh-17945 will allow us to set a different seed so that the numbers that render in the test make more sense.

You can't reproduce these values by running the code because gh-13863 also removed the visible seed from the documentation. The numbers you get when running the example code locally will be stochastic, but the conclusion of the test will be correct most of the time you run the example.

Still, this mismatch between the numbers in the documentation and the ones generated by the code keeps coming up (e.g. gh-14313, gh-15644, gh-15696) and I'm not sure what the ultimate resolution should be.

• We could revert to showing random seeds in documentation, but this is contentious.
• We can avoid specifying numbers in the text of the documentation and just refer to "large" or "small" values of the statistic/p-value, but there is no guarantee that these descriptions will agree with the output when a user runs the code locally.
• We can hard-code data in the documentation, but this distracts from the purpose of the examples.
• We could document that numbers generated by the code will be stochastic, but this would get repetitive unless it only appears in the tutorials, in which case it would often be missed.

@tupui, what do you think?

[tupui]

I would just document once that such things are stochastic and in the text don't refer to fixed values.
I would close this issue. Nothing extra to keep track that we don't already track or know.

[mdhaber]

We can close it when the example is fixed in gh-17945.