DOC: stats.bootstrap: improve documentation multidimensional input

[aangelopoulos]

Hello! I am writing to describe some errors in the bootstrap function. I am happy to help fix these errors if we agree that they are indeed bugs.

The issue arises when you give the bootstrap a statistic which depends on $X$ and $y$. Say $X \in \mathbb{R}^{n \times d}$ and $y \in \mathbb{R}^n$. (This follows standard scipy conventions, as in the logistic regression implementation). Let's say also that the statistic depends on both $X$ and $y$; for example, let's say the statistic is $\mathbf{1}^{\top}X^{\top}y$. Then we would expect running bootstrap([X,y], statistic, n_resamples=7, paired=True) to give us an array of size 7; it should just resample $X$ and $y$ 7 times, and calculate the statistic for each resampling.

However, this is not what happens. In this MWE below, the function fails to run. Errors in the resampling protocol are causing this bug.

1. Firstly, to be consistent with the rest of scipy, we should have the first column index the length of the sample, and the rest of the columns indexx dimensions.
2. Running _bootstrap_resample(X, n_resamples=7) returns an array with shape (n, n_resamples, d). Running _bootstrap_resample(y, n_resamples=7) gives an array of shape (n_resamples,n).
3. Calculating `statistic(*resampled_data, axis=-1)' will fail, due to a dimension mismatch. In some other examples, it will fail silently, which can be worse, causing errors that propagate into downstream statistical analyses.

Thank you for your consideration!

 import numpy as np
 from scipy.stats import bootstrap
 
 def statistic(x,y):
     v = np.ones((2,))
     vXt = v@x.T
     return vXt@y
 
 X = np.column_stack((0*np.ones((10,)), np.ones((10,))))
 Y = 100*np.ones((10,))
 data = [X,Y]
 n_resamples = 100
 statistic_original = statistic(*data)
 print("Original statistic (working shape): ", statistic_original)
 print("The following line fails due to bad shape!")
 bsd = bootstrap([X,Y], statistic=statistic, n_resamples=15, paired=True).bootstrap_distribution
 print(bsd, bsd.shape)

[mdhaber]

This doesn't look like a bug at first glance. X gets broadcasted against y along all dimensions except ignoring axis. Were you expecting them to be broadcasted before performing the operations?

[aangelopoulos]

Would you mind elaborating? I'm not sure if I understand your point.

Additionally, I'm not sure if we're agreeing on the expected behavior. To me, the expected behavior would be that if $X$ has dimension (n,d) and $Y$ has dimension (n,), if I run

bootstrap([X,Y], statistic, n_resamples=15, paired=True).bootstrap_distribution

then it should output 15 bootstrap samples of statistic(_X,_Y), where _X is an (n,d) bootstrap sample of X, and _Y is an (n,) bootstrap sample of Y, and the same rows are selected for both _X and _Y.

Do you agree that this should be the expected behavior?

Thanks for your quick response!

[mdhaber]

If I understand correctly, the way to achieve the behavior you're after is to make statistic a function of an index i:

def statistic(i):
    x = X[i]
    y = Y[i]
    ...  # whatever you want to do

and pass i = np.arange(n) as your "data".

i = np.arange(10)
bsd = bootstrap((i,), statistic=statistic, n_resamples=15)

Each time statistic is called, it will be passed an array of indices (sampled with replacement), and your statistic uses these indices to select the rows of X and elements of Y and compute the statistic.

That said, there are some documentation improvements and possibly input validation improvements to be made here, so I'll leave this issue open until I get to them shortly. Actually, bootstrap is not doing exactly what I was expecting, either - I was expecting bootstrap to broadcast the arrays of the input data before getting started. (Almost every other scipy.stats reducing function does so at this point¹, including monte_carlo_test and permutation_test, which are in the same file.) If it had, you would have gotten an error earlier that would have made it more obvious that the function was not being used as intended. I'll look into that.

As for why the function does not behave as you were expecting, it's because the function is primarily designed for statistics that accept multiple scalar-valued samples. An example is the t test. Suppose I'm doing an experiment and I want to compare both X[0] and X[1] (separately) against another group, Y. (Disclaimer to the statistics police: Yes, Dunnett's test may be more appropriate, but let's assume I'll use false_discovery_control to adjust the p-values.)

from scipy import stats
x = [[1, 2, 3, 4, 5],
        [6, 7, 8, 9, 0]]
y = [10, 11, 12]
stats.ttest_ind(x, y, axis=1)
# TtestResult(statistic=array([-7.74596669, -2.32565127]), pvalue=array([0.00024326, 0.0589908 ]), df=array([6., 6.]))

ttest_ind will effectively begin by "broadcasting" y to the shape of x except along axis.

y = [[10, 11, 12],
     [10, 11, 12]]

Then it will effectively perform two separate tests with corresponding slices of X and Y:

stats.ttest_ind(x[0], y[0])
# TtestResult(statistic=-7.745966692414835, pvalue=0.00024325611128491657, df=6.0)
stats.ttest_ind(x[1], y[1])
# TtestResult(statistic=-2.325651273547659, pvalue=0.05899079663707629, df=6.0)

That is the sort of thing bootstrap is designed to do when it gets n-dimensional arrays for the samples and an axis argument.

However, you appear to be thinking in terms of a vector-valued sample in which each observation is an entire row of X. You can work with these sorts of statistic by converting them as above and thinking of the index i as the sample.

Footnotes

1. Besides those that are explicitly multidimensional, like wasserstein_distance_nd ↩

[aangelopoulos]

Got it. Thanks for all the detailed help, @mdhaber. I do agree that, if the bootstrap function remains the same, the documentation should be updated to reflect these behaviors.

I'd also like to make the case that extending this function to naturally handle multi-input statistics would be a big improvement. The bootstrap is normally used to quantify the variation in multidimensional-input statistics, the main example being regressions. If I plug in LogisticRegression().fit as the statistic, it should be able to give me the variation in the logistic regression solution, and so on.

Modifying the function to have this default behavior would make it substantially more useful, and it would be equally simple. It would also have a simpler user interface that wouldn't result in confusion: you just put in [X,y] as your data, a statistic that accepts X and y, and bootstrap intervals/samples will be generated.

Below is some initial code for generating bootstrap resamples from statistics with arbitrary inputs. This approach is equally computationally efficient as the current code in the case of scalar-input statistics, but it is more flexible. It also expects input in an $n \times d$ array (which is more standard as opposed to the current $d \times n$). I'd be happy to work with you to integrate an approach like this into scipy. I think this software is very important, and that millions of people will end up interacting with bootstrap.

What do you think?

def bootstrap(data, statistic, n_resamples):
    """
    Bootstrap the given statistic on the data.

    Args:
        data (ndarray, list): Data to bootstrap. First dimension is the number of observations. Can be an array or a list of arrays, in which case the resamples will be paired.
        statistic (callable): Statistic to compute. Should take data as input if data is a single array, or *data if data is a list of arrays.
        n_resamples (int): Number of bootstrap resamples.

    Returns:
        ndarray: Bootstrap resamples of the statistic.
    """
    bootstrap_distribution = []
    if not isinstance(data, list):
        data = [data]
    for i in range(n_resamples):
        resample_indexes = np.random.choice(
            data[0].shape[0], data[0].shape[0], replace=True
        )
        resample_data = [d[resample_indexes] for d in data]
        bootstrap_distribution.append(statistic(*resample_data))
    return np.array(bootstrap_distribution)

[mdhaber]

It also expects input in an $n \times d$ array (which is more standard as opposed to the current $d \times n$).

I'm not sure what you mean by changing what bootstrap expects. First, we wouldn't want to make a backward-incompatible change. Moreover, bootstrap doesn't expect much - the input arrays can be of any number of dimensions and shapes as long as they are broadcastable according to standard conventions - actually, this requirement is even more relaxed because they need not be broadcastable along axis.

This approach is equally computationally efficient as the current code in the case of scalar-input statistics, but it is more flexible.

I don't think so. It looks like there is a loop iteration for each resample. When you want to do ~10k bootstrap samples (current default), this would be very slow. For example:

import numpy as np
from scipy import stats
# define your bootstrap function here
rng = np.random.default_rng(3248924234)
n_resamples = 9999
x = rng.random(100)
%timeit bootstrap([x,], np.mean, n_resamples)
# 431 ms ± 54.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit stats.bootstrap((x,), np.mean, n_resamples=n_resamples)
# 19 ms ± 429 µs per loop (mean ± std. dev. of 7 runs, 10 loops each

It would need to allow for doing all the resamples at once - or in groups with sizes given by the batch parameter - and make use of an axis argument in the statistic function. We'll also be adding array API support so that bootstrap can be used with CuPy, JAX, PyTorch, etc... arrays soon, so that needs to be considered.

I point this out not because it's hard to generalize, but it's something else you need to consider about the current code before we can consider a suggestion about changing it.

---

I still think documentation is the best way to accomodate the large number of possible behaviors users might have for bootstrap. There is a fine balance between accomodating all possible behaviors users might want and keeping the interface simple. Here we have a case where it's quite simple to get the behavior you wanted with minimal changes to your code. But if you wanted to consider changing something about bootstrap, look at how the paired works. It doesn't modify the main code at all, it's handled entirely in the input validation/standardization function by making data an index and wrapping the user-provided statistic to index the original data before performing the calculation. If we were to change anything, it would probably do something like that.

[aangelopoulos]

look at how the paired works. It doesn't modify the main code at all, it's handled entirely in the input validation/standardization function by making data an index and wrapping the user-provided statistic to index the original data before performing the calculation. If we were to change anything, it would probably do something like that.

If one wanted to make minimal changes to the current function, this makes sense to me; an argument like multi_d_input could be added, and then the solution you gave in the previous comment could be run.

It looks like there is a loop iteration for each resample.

True, but it's easy to parallelize this; I was trying to give the simple case where batch=1, in which case this reduces to the current code. We can always batch this to improve efficiency. But it sounds like, if you want this code to be backwards compatible, that the only way to do that would be adding a new flag as you suggested.

[mdhaber]

So how would the multi_d_input work? How would it interact paired, in particular, and also axis in cases where the arrays are >2d? Or would it be restricted for use in certain cases?

[aangelopoulos]

It's going to take more thought, but here's a first cut. Let me know if I've misunderstood something.

When multi_d_input==True and paired==True, we consider data to be a list that contains a set of inputs with paired rows, and statistic to be any statistic for which statistic(*data) will run. Then, the expected behavior will be to generate n_resamples copies of statistic(*resampled_data). The argument axis is ignored when multi_d_input==True. If the user wants to compute the statistic differently, they can change the definition of the statistic outside the bootstrap function.

Also, quick question:

It would need to allow for doing all the resamples at once - or in groups with sizes given by the batch parameter - and make use of an axis argument in the statistic function.

It doesn't look like the current code does this. Statistic never takes axis as an argument. Do you mean that this is done implicitly because of the broadcasting? (Not sure how this would work, since we always have axis=-1 in the current code.)

[mdhaber]

No, the code currently moves axis to -1 and always passes -1 to the statistic. Statistics that don't already accept an axis argument are wrapped to accept one.

The sort-of-bug exposed by this issue is that if axis is moved before the arrays are broadcasted, that is inconsistent with how other functions of this sort work. (This was written before I wrote _broadcast_arrays that accepts an axis argument.) I have a plan to address this separately in a backward-compatible way, so let's not worry about that here.

[aangelopoulos]

Got it. Thanks for explaining. And separately from the axis question, what do you think of adding a multi_d_input argument that behaves as above?

[mdhaber]

I agree that this is a very important use case. Still, I would prefer to start with documentation for these reasons:

• bootstrap has been out for a few years, and this is the first request of the sort, so working on this right now doesn't make it to the top of my queue, personally. Please take a look at the other open issues and PRs. There is a lot going on. You're welcome to try your luck with other reviewers; I'm just saying that I would prefer not to review a PR in this direction right now.

• The additional option - which we'd have to think through (e.g. names are always hard, and are there unintented interactions with other arguments?), document, and test carefully, then maintain in perpetuity - doesn't look to me like it would save the user much trouble in the scheme of things. The same effect can be achieved with changes in two or three lines of user code as shown above.¹.

• Even together, the convenience arguments paired and multi_d_input would not cover all cases of this sort. Whatever we do, it is worth adding a recipe of this sort to the documentation anyway so users can handle even more exotic cases (e.g. some samples paired, others independent).

• As proposed, multi_d_input is incompatible with the axis argument. Perhaps incompatible options are not always a bad thing, but they can be a sign that too much is being shoved into a signature.

• Moreover, bootstrap already has four conceptually different kinds of array dimensions to consider:

  • The "core" dimension along which the statistic is calculated / gets reduced away, as specified by the axis argument
  • The "batch" dimensions in the samples provided by the user (dimensions other than the core dimension)
  • Additional, tempory batch dimensions used for resampling
  • Additional dimensions for multi-output/vector-valued statistics

  This is quite complicated already, and I would prefer not to add another that is incompatible with some of the rest. Besides, SciPy currently has relatively few statistics / hypothesis tests that support multi-dimensional observations (wasserstein_distance_nd, multivariate_normal.fit, and directional_stats are all quite recent exceptions) and the interface for working with such things along with the other sorts of axes above is not very well established. I have felt for a while that we need another type of axis argument or a very standard convention to work with them properly, but we don't have either yet, so this is going in a murky area that we're just not ready for when considering scipy.stats as a unified whole.

• Conversion of bootstrap to be array API compatible is imminent (in SciPy terms... like within the next year); see ENH: stats: add array API-support #20544 to see the rapid progress in this area. I don't want to add anything right now that could complicate that even slightly.

However, I'd be happy to review a PR that adds notes and/or and example to the documentation. We might even be able to get such a change into SciPy 1.14.0 (which will be released shortly). If you'd like to go that route, please draft it here and then we can open a PR.

Footnotes

1. The same can be said of paired, but that was a case that was anticipated in advance and clearly needed to work with existing SciPy functions (e.g. pearsonr and other correlation functions). ↩