TST: stats: `check_sample_var` should be two-sided (Trac #1546)

[scipy-gitbot]

Original ticket http://projects.scipy.org/scipy/ticket/1546 on 2011-10-30 by @josef-pkt, assigned to unknown.

check_sample_var doesn't test anything

see gh-2070

[scipy-gitbot]

@josef-pkt wrote on 2011-10-30

This is what the test is supposed to do, I don't think we have this test in scipy.stats
sample variance on top (check ddof of variance, not that it matters at large sample sized)

http://en.wikipedia.org/wiki/Chi-squared_test#Chi-squared_test_for_variance_in_a_normal_population
http://en.wikipedia.org/wiki/Variance#Distribution_of_the_sample_variance

I think the distribution should be correct also for large samples if the distribution is not normal, (central limit theorem?).

edit (2013-08-23 after starting to look at variance tests for statsmodels)
test on variance are very sensitive to larger deviations from normality
small sample hypothesis test results can be strongly biased (not correctly sized)

[scipy-gitbot]

@josef-pkt wrote on 2011-10-30

Doing the obvious correction of the typo still didn't catch tukeylambda.
Finally, I realized that we need two sided tests, even though it was mentioned in the docstring. (It's not like tests of nested models that only need to check one side.)

my current version, edited on top of scipy 0.9, is

def check_sample_var(sv,n, popvar):
    '''
two-sided chisquare test for sample variance equal to hypothesized variance
    '''

    df = n-1
    chi2 = (n-1)*sv/float(popvar)
    #pval = stats.chisqprob(chi2,df)*2
    pval = min(stats.chi2.cdf(chi2,df), stats.chi2.sf(chi2,df)) #need two-sided
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))

and has with the very low pvalue 0.005 the following failures (there are also 12 failures at pval=0.01) with the not-slow option

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.fatiguelife_gen object at 0x02F118F0>, (29,), array(421.5), array(884942.25), 381.5231692182964, 659709.81429483229, 1000, 'fatiguelifesample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 744.737981, 0.000000, v,sv=884942.250000,659709.814295

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.gamma_gen object at 0x02F1A130>, (1.9932305483800778,), array(1.9932305483800778), array(1.9932305483800778), 1.9049475453784119, 1.683287232178736, 1000, 'gammasample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 843.657522, 0.000127, v,sv=1.993231,1.683287

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.genlogistic_gen object at 0x02F11EB0>, (0.41192440799679475,), array(-1.8996417990533176), array(8.5520107632574316), -1.6915023776411073, 7.2701839656308955, 1000, 'genlogisticsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 849.263873, 0.000217, v,sv=8.552011,7.270184

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.gilbrat_gen object at 0x02F234B0>, (), array(1.6487212707001282), array(4.670774270471604), 1.5542857947354762, 3.0177319205419764, 1000, 'gilbratsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 645.442065, 0.000000, v,sv=4.670774,3.017732

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.gumbel_l_gen object at 0x02F1A590>, (), array(-0.57721566490153287), array(1.6449340668482264), -0.48770832402349951, 1.403145247583542, 1000, 'gumbel_lsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 852.157014, 0.000284, v,sv=1.644934,1.403145

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.invgamma_gen object at 0x02F1A9B0>, (2.0668996136993067,), array(0.93729530609975309), array(13.131951625091878), 0.98003361245261933, 2.2163927217803887, 1000, 'invgammasample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 168.609845, 0.000000, v,sv=13.131952,2.216393

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.loggamma_gen object at 0x02F23270>, (0.41411931826052117,), array(-2.4617679388246936), array(6.8426502245961007), -2.4057296757832174, 5.9831621683265688, 1000, 'loggammasample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 873.518127, 0.001755, v,sv=6.842650,5.983162

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.lognorm_gen object at 0x02F233F0>, (0.95368226960575331,), array(1.5757871665018548), array(3.6827062109137709), 1.4935980843433927, 2.4720855647743383, 1000, 'lognormsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 670.597473, 0.000000, v,sv=3.682706,2.472086

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.pareto_gen object at 0x02F23AD0>, (2.621716532144454,), array(1.6166305764162521), array(1.6034057205287133), 1.6550137301062546, 1.2674974417903764, 1000, 'paretosample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 789.712752, 0.000000, v,sv=1.603406,1.267497

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.powerlaw_gen object at 0x02F23C70>, (1.6591133289905851,), array(0.62393479469353574), array(0.85857496135780687), 0.63812341708244613, 0.061503031152340903, 1000, 'powerlawsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 71.562217, 0.000000, v,sv=0.858575,0.061503

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.tukeylambda_gen object at 0x02FD6590>, (3.1321477856738267,), array(0.0), array(0.30476472279111871), 0.0090800061342219563, 0.026536393118696645, 1000, 'tukeylambdasample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 86.984663, 0.000000, v,sv=0.304765,0.026536

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.tukeylambda_gen object at 0x02FD6590>, (0.14000000000000001,), array(0.0), array(24.379664446968761), 0.090612433601054559, 2.0217342128370261, 1000, 'tukeylambdasample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.005, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 82.844146, 0.000000, v,sv=24.379664,2.021734

----------------------------------------------------------------------
Ran 1149 tests in 10.499s

FAILED (SKIP=1, failures=12)

3 more failures in the "slow" distributions

======================================================================
FAIL: test_continuous_basic.test_cont_basic_slow(<scipy.stats.distributions.johnsonsu_gen object at 0x02DFFE10>, (2.554395574161155, 2.2482281679651965), array(-1.5421538673934436), array(0.76298819698253784), -1.4802090580087397, 0.64030549378085866, 1000, 'johnsonsusample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 838.368393, 0.000075, v,sv=0.762988,0.640305

======================================================================
FAIL: test_continuous_basic.test_cont_basic_slow(<scipy.stats.distributions.rdist_gen object at 0x02E07ED0>, (0.90000000000000002,), array(0.0), array(1.7800284850122077), 0.034195658437930029, 0.53053935902899019, 1000, 'rdistsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 297.752999, 0.000000, v,sv=1.780028,0.530539

======================================================================
FAIL: test_continuous_basic.test_cont_basic_slow(<scipy.stats.distributions.recipinvgauss_gen object
at 0x02ECD1F0>, (0.63004267809369119,), array(2.5871940660171391), array(3.5871940659111603), 2.4895378659294742, 3.0462804278153826, 1000, 'recipinvgausssample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose-1.0.0-py2.6.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 164, in wrap
    return func(*a, **kw)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 259, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Python26\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Python26\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 848.360610, 0.000199, v,sv=3.587194,3.046280

----------------------------------------------------------------------
Ran 1576 tests in 169.179s

FAILED (SKIP=1, failures=15)

[scipy-gitbot]

@josef-pkt wrote on 2011-10-31

I fixed the test in my fork, but running the test suite still has 8 failures in the "not-slow" distributions, possibly false alarms. The numbers don't look very far off.
(I didn't change anything in the slow distributions.)

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.fatiguelife_gen object at 0x02E9B190>, (0.75,), array(1.28125), array(0.9580078125), 1.2487700710120595, 0.81356018397341889, 1000, 'fatiguelifesample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 165, in wrap
    return func(*a, **kw)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 260, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 848.371603, 0.000199, v,sv=0.958008,0.813560

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.gamma_gen object at 0x02E9BA10>, (1.9932305483800778,), array(1.9932305483800778), array(1.9932305483800778), 1.9049475453784119, 1.683287232178736, 1000, 'gammasample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 165, in wrap
    return func(*a, **kw)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 260, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 843.657522, 0.000127, v,sv=1.993231,1.683287

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.genlogistic_gen object at 0x02E9B6D0>, (0.41192440799679475,), array(-1.8996417990533176), array(8.552010763257432), -1.6915023776411073, 7.2701839656308955, 1000, 'genlogisticsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 165, in wrap
    return func(*a, **kw)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 260, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 849.263873, 0.000217, v,sv=8.552011,7.270184

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.gilbrat_gen object at 0x02F58BB0>, (), array(1.6487212707001282), array(4.670774270471604), 1.5542857947354762, 3.0177319205419764, 1000, 'gilbratsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 165, in wrap
    return func(*a, **kw)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 260, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 645.442065, 0.000000, v,sv=4.670774,3.017732

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.gumbel_l_gen object at 0x02E9BC70>, (), array(-0.5772156649015329), array(1.6449340668482264), -0.48770832402349951, 1.403145247583542, 1000, 'gumbel_lsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 165, in wrap
    return func(*a, **kw)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 260, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 852.157014, 0.000284, v,sv=1.644934,1.403145

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.loggamma_gen object at 0x02F58850>, (0.4141193182605212,), array(-2.4617679388246936), array(6.842650224596101), -2.4057296757832174, 5.9831621683265688, 1000, 'loggammasample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 165, in wrap
    return func(*a, **kw)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 260, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 873.518127, 0.001755, v,sv=6.842650,5.983162

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.lognorm_gen object at 0x02F589F0>, (0.9536822696057533,), array(1.5757871665018548), array(3.682706210913771), 1.4935980843433927,
2.4720855647743383, 1000, 'lognormsample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 165, in wrap
    return func(*a, **kw)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 260, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 670.597473, 0.000000, v,sv=3.682706,2.472086

======================================================================
FAIL: test_continuous_basic.test_cont_basic(<scipy.stats.distributions.pareto_gen object at 0x02F610F0>, (2.621716532144454,), array(1.616630576416252), array(1.6034057205287133), 1.6550137301062546, 1.2674974417903764, 1000, 'paretosample mean test')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\programs\python27\lib\site-packages\nose-1.0.0-py2.7.egg\nose\case.py", line 187, in runTest
    self.test(*self.arg)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 165, in wrap
    return func(*a, **kw)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 260, in check_sample_meanvar_
    check_sample_var(sv, sn, v)
  File "C:\Programs\Python27\Lib\site-packages\scipy\stats\tests\test_continuous_basic.py", line 292, in check_sample_var
    npt.assert_(pval > 0.01, 'var fail, t,pval = %f, %f, v,sv=%f,%f' % (chi2,pval,popvar,sv))
  File "C:\Programs\Python27\lib\site-packages\numpy\testing\utils.py", line 34, in assert_
    raise AssertionError(msg)
AssertionError: var fail, t,pval = 789.712752, 0.000000, v,sv=1.603406,1.267497

----------------------------------------------------------------------
Ran 1160 tests in 10.212s

FAILED (SKIP=1, failures=8)

[WarrenWeckesser]

@josef-pkt, can we close this? After looking at the code and at the links you gave, I noticed a mistake in check_sample_var, and I created a PR with a fix: gh-13167.

I don't know if that bug was the source of the problems seven years ago, and I don't know if the problems that you reported are still relevant. The tests in the PR all pass, and as far as I can tell, the problem no longer exists.

[mdhaber]

This should stay open until the test is corrected or removed. The typo fixed in gh-13167 was noted above, but also noted above is that the test performed is one-sided: it only fails the assertion if the sample variance is too high. The code above fixed the test, but it sounds like it didn't turn into a PR because some distributions failed the test.

A better reference for the intended test is here.

I will open a PR to fix the test, and we can decide what to do about the failing distributions. For instance (I haven't checked carefully), it may be acceptable to xfail because the test failure may be due to violation of test assumptions rather than an unusual sample variance. The Wikipedia reference given above suggests that the test is derived for samples from a normal distribution, and while it may be relatively robust to different distributions, we have a lot of distributions, so it may not work well for all of them. As noted above:

edit (2013-08-23 after starting to look at variance tests for statsmodels)
test on variance are very sensitive to larger deviations from normality
small sample hypothesis test results can be strongly biased (not correctly sized)

We might just decide to drop the test altogether. It may be too prone to false positives to be valuable.