BUG: stats.ttest_ind returns wrong p-values with permutations

[jmkuebler]

Describe your issue.

When running the two-sample t-test with permutations we do not make any assumptions about the distributions.
A user would reject the two-sample null hypothesis when obtaining a p-value smaller or equal to their significance level and we should guarantee that under the null hypothesis this does not happen with probability larger than the significance level.
Hence, the p-value should never be 0!

But with the current implementation it does.

Note that if a large number of permutations is used, this does not lead to a problem. But an inexperienced user might wrongly reject their null-hypothesis, because they used few permutations and received pvalue equal 0.

Reproducing Code Example

from scipy import stats
import numpy as np

rng = np.random.default_rng()

for i in range(200):
  rvs1 = stats.norm.rvs(loc=5, scale=10, size=500, random_state=rng)
  rvs2 = stats.norm.rvs(loc=5, scale=10, size=500, random_state=rng)
  print(stats.ttest_ind(rvs1, rvs2, permutations=1)[1])

Error message

N/A

SciPy/NumPy/Python version information

1.7.3 1.21.6 sys.version_info(major=3, minor=7, micro=13, releaselevel='final', serial=0)

[jmkuebler]

@mortonjt maybe you can have a look here. I think you committed the original code.

[mdhaber]

This is correct in stats.permutation_test. References that support the need to add +1 in the numerator and denominator for randomized tests are [2] and [3] of the documentation.

I'll have to look back at why I didn't already fix this in the permutation t-test in particular, but we should.

[jmkuebler]

@mdhaber
Maybe the reason was that if the set number of permutations is larger than the number of different permutations, the method takes a deterministic approach.
In this case, adding +1 is not necessary and should actually not be done!
Therefore, one additionally has to check whether this was the case.
Furthermore we have to adjust the hard-coded expected p-values for the tests.
I have done that for some (not all yet) in my PR #16460