scipy.statsbinned_statistic_2d: incorrect binnumbers returned

[pbranson]

For certain inputs binned_statistic_2d returns incorrect bin numbers

xEdges = np.arange(79950.,500050.,100.)
yEdges = np.arange(7489950.,7860050.,100.)
x = 356643.378
y = 7813944.500
binned, xedges, yedges, binnums = binned_statistic_2d((x,), (y,), (0.5,), 'mean', bins=[xEdges,yEdges],expand_binnumbers=True)

The binnums seem to be incorrect:
> binnums Returns: array([[3678], [1291]])
> np.where(np.isfinite(binned)) Returns: (array([2766]), array([3239]))

I worked around this using the following code to calculate the bin numbers:

x_inds = np.searchsorted(xEdges, x)
y_inds = np.searchsorted(yEdges , y)

[lzkelley]

The actual binning still seems to be done properly in this case, it's only the bin-numbers which are failing. Indeed, this can be shown even without setting expand_binnumbers=True:

Following @jakevdp example from the previous (linked) issue

    # `binnumbers` work for this:
    # x = [0.1, 0.1, 0.1, 0.6]
    # y = [2.1, 2.6, 2.1, 2.1]
    # xe = [0.0, 0.5, 1.0]
    # ye = [2.0, 2.5, 3.0]

    # `binnumbers` fail for this:
    x = [356643.378, 357643.378]
    y = [7813944.500, 7814944.500]
    xe = np.arange(79950.0, 500050.0, 100.0)
    ye = np.arange(7489950.0, 7860050.0, 100.0)

    stats = sp.stats.binned_statistic_2d(x, y, np.arange(len(x)), 'count', bins=[xe,ye])
    x_ind, y_ind = np.unravel_index(stats.binnumber,
                                    (len(stats.x_edge) + 1, len(stats.y_edge) + 1))
    print("x:")
    print(x_ind)
    print(np.digitize(x, xe))
    # x:
    # [3678 3689]
    # [2767 2777]

    print("y:")
    print(y_ind)
    print(np.digitize(y, ye))
    # y:
    # [1291 2599]
    # [3240 3250]

The key section of code is here, but unfortunately this is above my pay-grade and I just don't really understand what it's doing... regardless, since the bulk of the code is actually still working I think there are two possibilities:

1. 'unravel' the binnumbers indices in a different way (i.e. instead of np.unravel_index) which works in all use cases, or
2. rewrite the code to use a different indexing pattern, perhaps using np.ravel_multi_index.

I'm hoping someone wiser (hence pinging jakevdp) will chime in on this. Sorry it's taking a while @pbranson

[pbranson]

Something that I found a little counter-intuitive (to me anyway!) is that the expanded bin numbers have a zeroth index for data less than the smallest bin - this results in the indexes being offset by 1 compared to the index of the bin in the 2D statistic array to which the data was allocated. Could it possible return -1 and len(array) for data outside the range of the bin edges?

…

On Sun, Feb 12, 2017 at 12:32 PM, Luke ***@***.***> wrote: The actual binning still seems to be done properly in this case, it's only the bin-numbers which are failing. Indeed, this can be shown even without setting expand_binnumbers=True: Following @jakevdp <https://github.com/jakevdp> example from the previous (linked) issue <#5449 (comment)> # `binnumbers` work for this: # x = [0.1, 0.1, 0.1, 0.6] # y = [2.1, 2.6, 2.1, 2.1] # xe = [0.0, 0.5, 1.0] # ye = [2.0, 2.5, 3.0] # `binnumbers` fail for this: x = [356643.378, 357643.378] y = [7813944.500, 7814944.500] xe = np.arange(79950.0, 500050.0, 100.0) ye = np.arange(7489950.0, 7860050.0, 100.0) stats = sp.stats.binned_statistic_2d(x, y, np.arange(len(x)), 'count', bins=[xe,ye]) x_ind, y_ind = np.unravel_index(stats.binnumber, (len(stats.x_edge) + 1, len(stats.y_edge) + 1)) print("x:") print(x_ind) print(np.digitize(x, xe)) # x: # [3678 3689] # [2767 2777] print("y:") print(y_ind) print(np.digitize(y, ye)) # y: # [1291 2599] # [3240 3250] The key section of code is here <https://github.com/scipy/scipy/blob/41baa3be6b19a6321912e501e24b086518cd6e7b/scipy/stats/_binned_statistic.py#L543-L549>, but unfortunately this is above my pay-grade and I just don't really understand what it's doing... regardless, since the bulk of the code *is actually still working* I think there are two possibilities: 1. 'unravel' the binnumbers indices in a different way (i.e. instead of np.unravel_index) which works in all use cases, or 2. rewrite the code to use a different indexing pattern, perhaps using np.ravel_multi_index. I'm hoping someone wiser (hence pinging jakevdp) will chime in on this. Sorry it's taking a while @pbranson <https://github.com/pbranson> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7010 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AM3bQE0Lt6fFkks2sDB2yCUahu_ApR11ks5rbotjgaJpZM4L5Fwj> .

[solarjoe]

Not sure if this the same issue or if I just got something wrong.
I took the example from the binned_statistic_2d docstring and expanded it by an outlier.

x = [-0.1, 0.1, 0.1, 0.1, 0.6]
y = [1.1, 2.1, 2.6, 2.1, 2.1]
binx = [0.0, 0.5, 1.0]
biny = [2.0, 2.5, 3.0]
ret = stats.binned_statistic_2d(x, y, None, 'count', bins=[binx,biny])
print(ret.binnumber)
ret = stats.binned_statistic_2d(x, y, None, 'count', bins=[binx,biny], expand_binnumbers=True)
print(ret.binnumber)

>>> [0 5 6 5 9]
>>> [[0 1 1 1 2]
     [0 1 2 1 1]]

The expanded indices are correct, but how to the non-expanded work?

The first index (0) is the outlier, but since the binning grid only is 2 x 2, why are the other indices 5 6 5 9

I saw this post #5449 (comment) and thought I understood how the un-ravelling of the indices works. But maybe I am wrong.

@pbranson, true, it's not intuitive. I guess the behavior is kept similar to numpy.digizite. Its docstr says

If values in x are beyond the bounds of bins, 0 or len(bins) is returned as appropriate.

At least this should also be added to the docstring of the binned_statistic functions.

[yhlam]

I see similar issue.

Test code:

import numpy as np
import scipy.stats

x = np.array([1, 2, 3, 4])
y = np.array([1, 2, 3, 4])
bins = np.array([
    np.linspace(0.5, 4.5, 5),
    np.linspace(0.5, 4.5, 3),
])
count, x_edges, y_edges, binnum = scipy.stats.binned_statistic_2d(
    x, y, None, 'count', bins=bins,
    expand_binnumbers=True
)
print('x: ', x)
print('y: ', y)
print('X Edges:', x_edges)
print('Y Edges:', y_edges)
print('Bin number:')
print(binnum)

Output:

x:  [1 2 3 4]
y:  [1 2 3 4]
X Edges: [ 0.5  1.5  2.5  3.5  4.5]
Y Edges: [ 0.5  2.5  4.5]
Bin number:
[[1 2 3 4]
 [3 0 3 0]]

The expected bin number should be

[[1 2 3 4]
 [1 1 2 2]]

I believe the problem is the binnumbers is encoded and decoded by different order of dimensions.

The binnumbers is encoded by the sorted dimensions. In my example, the number of bins including the outliner buckets is (6, 4), but the binnumbers is encoded in the sorted dimension, which is (4, 6) instead. I guess this is attempt to reduce the size of the encoded binnumbers. This is not really a problem, as long as the binnumbers is also decoded by (4, 6) rather than (6, 4). Unfortunately, it is not case.

When the binnumbers is expanded, it is expanded by the original dimension order, i.e. (6, 4) rather than (4, 6) in my example. As the binnumbers is encoded and decoded with different dimensions, the expanded value is completely wrong.

I am going to submit a pull request on this.

[solarjoe]

Could you also add something to testing? I wonder why this was not
caught earlier.

[yhlam]

@solarjoe I updated test cases in my pull request #7423. The original test case has the same number of bins in all dimensions, so the order of dimension doesn't matter. The original code without my fix would fail in the updated test case.

[yhlam]

Actually, the more I think about it, the more I don't understand why the dimensions needed sorted in the binnumber logic. It appears to me to be some sort of optimization, but I can't really think of any rationale behind. That logic has been there since 2012, when the binned statistic logic committed to scipy.

If there is no good reason to sort the dimensions, the binnumber logic can be replaced by np.ravel_multi_index() and the reordering logic can be removed. I think this will simplify the complexity of the code. This may be even slightly more efficient as np.ravel_multi_index() is implemented in C, although I haven't done any profiling yet.

One drawback is that the unexpanded binnumber would change if the input bin dimensions is not monotonic increasing, since the current binnumber is encoded by the sorted dimensions. But this is not documented and not intuitive, I doubt that if there is any people depending on this behavior. The current behavior may actually be incorrect as the documentation of 0.19.0 says:

The returned binnumber is a shape (N,) array of linearized indices mapping each element of sample to its corresponding bin (using row-major ordering)

The current logic actually uses row-major ordering after swapping axis to the shape of np.sort(shape)

I can submit another pull request to simplify the logic. Let me know if I should start a new issue or pull request to discuss.

Thanks

[lzkelley]

I totally agree that the reordering seems unnecessarily complex. It seems like it would be great to remove that. If people are worried about the performance, we could always just benchmark both methods to confirm...

[yhlam]

My pull request is updated. I did a simple profiling on this change, there is no obvious performance change.

Testing code:

import timeit

import numpy as np
from scipy.stats import binned_statistic_2d


if __name__ == '__main__':
    np.random.seed(42)
    x = np.random.random(100000)
    y = np.random.random(100000)
    code = "binned_statistic_2d(x, y, None, 'count', bins=(10, 5))"

    for size in [10, 100, 1000, 10000, 100000]:
        globals = {
            'binned_statistic_2d': binned_statistic_2d,
            'x': x[:size], 'y': y[:size]
        }
        time = timeit.repeat(code, repeat=3, number=10000, globals=globals)
        print('size = {}:'.format(size), min(time))

My testing environment

Python version: 3.5.2
Numpy version: 1.12.1
OS: Linux Mint 18 (64bit)
Processor: Core i5-4460 @ 3.2GHz x 4
Memory: 8G

Output before the change:

size = 10: 1.7536323829999674
size = 100: 1.8471689620000689
size = 1000: 2.538789491999978
size = 10000: 7.255203272000017
size = 100000: 63.623585149000064

Output after the change:

size = 10: 1.533078791999742
size = 100: 1.6299732600000425
size = 1000: 2.322452835000149
size = 10000: 7.2526375130000815
size = 100000: 63.6260423599997