Statistics Review: variation (Trac #65)

[scipy-gitbot]

Original ticket http://projects.scipy.org/scipy/ticket/65 on 2006-04-02 by @rkern, assigned to @rkern.

The function variation in file source:trunk/Lib/stats/stats.py needs review.

Please look over the StatisticsReview guidelines and add your comments below.

[scipy-gitbot]

@rkern wrote on 2006-04-10

Should this use the biased or unbiased version of standard deviation?

[aeklant]

Empty input returns nan due to division between undefined values (There is a RuntimeWarning raised). This should be handled explicitly but I'm not sure whether it would be better to return np.nan or to raise a ValueError.

Thoughts?

[ev-br]

I'd suggest deprecate and remove in favor of a ratio of numpy equivalents. It's IMO easier to write a one-liner with explicit control over zero denominator, nan handling, empty input, and ddof correction than experiment with scipy.stats version to figure it out.

[rgommers]

From a quick search this is used quite a bit, for example in statsmodels (https://searchcode.com/codesearch/view/14872088/) and https://bitbucket.org/avalanche/avalanche (https://searchcode.com/codesearch/view/79218657/). And there's not a direct replacement in numpy. So -1 on deprecation.

[rgommers]

Empty input should return nan (without a warning). That's consistent with np.std([]) / np.mean([])

[rgommers]

Documentation needs some attention. It should clearly explain what this function is for in the first paragraph. https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-is-the-coefficient-of-variation/ and https://en.wikipedia.org/wiki/Coefficient_of_variation should be helpful here. In particular, point out that it doesn't make sense to use variation on symmetric distributions. The "ratio interval" terminology in the links above may be useful.

Regarding docstring format:

• Summary line needs to fit, now it stretches over 2 lines.
• Returns section is missing
• Example: might be nice to take two distributions that are only defined for x>0, one of which has variation <1 and one >1. The latter is then fat-tailed.

[rgommers]

There's a basic test; the axis argument needs to be tested though.

[hal-314]

Hi
I know that it's an old topic but I two days ago I need to compute the coefficient of variation (cv). I was surprised that scipy doesn't offers the option to compute it unbiased and uses the biased version of standard deviation. Pandas calculates the unbiased version of it by default (https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.std.html)

If you're agreed, I could do a PR that adds two options to control if the unbiased version of cv or/and standard deviation are wanted.

The function would be then:
variation(a, axis=0, nan_policy='propagate', unbiased = false, std_unbiased=false)
or
variation(a, axis=0, nan_policy='propagate', unbiased = false, ddof=1)

The unbiased cv is computed as (see https://en.wikipedia.org/wiki/Coefficient_of_variation#Estimation) :
cv_unbiased = (1 + 1/(4n))*cv_biased
where cv_biased is the current implementation.

[rgommers]

That seems reasonable. I'd prefer ddof=1 for consistency with other functions.

[cheese]

Also request a ddof argument for variation, just passing it through to std

[WarrenWeckesser]

(EDIT: I added another bullet to the list, with a comment about the correction 1 + 4/n.)

Here's my take on the remaining issues and enhancement requests (a couple are new from me):

• Add ddof parameter. I'm inclined to make the default ddof=0, to maintain backwards compatibility.
• It looks like several of Ralf's suggestions for improving the docstring have already been made, but a few more examples would be nice, including one that applies variation to a 2-d array with the use of axis, and another that uses nan_policy='omit' on an array that contains nan.
• variation([]) currently generates a sequence of five RuntimeWarnings and returns nan. This should be at most one warning, something like RuntimeWarning: variation of empty sequence. I think there should be a warning, because variation([]) is not a well-defined operation, and passing an empty sequence is probably a mistake. The problem is in the same category as np.mean([]), which also generates a warning.
• Unit tests for the case a.size == 0 are needed. These should include checking the behavior when axis is given with an n-d array (n > 1).
• Add a keepdims parameter. (This might be as simple as passing the given argument to the underlying std and mean functions.)
• Add an option for including the correction factor 1 + 4/n, as suggested by @hal-314 above. This, combined with ddof=1 (so the sample standard deviation is computed from the unbiased sample variance), gives a less biased estimate of the variation for normally distributed samples, according to https://en.wikipedia.org/wiki/Coefficient_of_variation#Estimation. (The correction factor doesn't actually eliminate the bias, but it does make it much smaller.)