BUG: KDTree balanced_tree is unbalanced for degenerate data

[peterbell10]

Reference issue

Fixes gh-14074

What does this implement/fix?

The balanced_tree argument is meant to create a more balanced tree by splitting at the median instead of the midpoint of the known bounds. However, the "sliding" part of the sliding midpoint rule was also being applied after choosing the median which results in an unbalanced split for degenerate data. Instead, keeping it as the exact median results in a balanced tree.

Additional information

The comments state the median rule comes from sklearn, and this is equivalent to KDTree code:
https://github.com/scikit-learn/scikit-learn/blob/3ae7c7615343bbd36acece57825d8b0d70fd9da4/sklearn/neighbors/_binary_tree.pxi#L1176-L1181

cc @sturlamolden @rainwoodman

[sturlamolden]

I think this PR is logically incorrect. The way our kd-tree works we cannot have the same value on both sides of the pivot (the splitting plane), which can happen in the case of ties. Our kd-tree cannot be exactly balanced in the presencence of ties. This is why we do data swapping after the partial sorting, when in theory all the swapping (partial sorting) should be completed. And then it follows that we need to consider sliding the splitting plane even when using the median as pivot. Otherwise we need to update all query methods to assume we can have equal values on both sides of a splitting plane. IIRC the kd-tree in sklearn does this, whereas ours does not because it started out using sliding midpoint and then added the median later on (mostly because it gave faster queries). I know this is annoying but I think the build method needs to be the way it is in order to be logically correct given what the query methods assume.

[peterbell10]

I think this PR is logically incorrect. The way our kd-tree works we cannot have the same value on both sides of the pivot

If this is true, then the sliding midpoint rule is broken as well. Consider constructing a node where all children are equal, but the tree is built with compact=False so it doesn't inspect the bounds and generate a leaf. All points are going to be on the same side of the midpoint and so the split will be chosen as either just after the first element or just before the last element in the list. Since all points are equal, then the same value must exist on both sides of the pivot.

[peterbell10]

If you agree with that, I should be able to fix both issues.

[sturlamolden]

It was constructed on the promise that
lesser < split <= greater.
You might be correct that the sliding midpoint is broken as well though, but I have never seen it break. Maybe it does not matter for the query functions. Or maybe someone has never constructed a bad enough data set and reported a failure. I am not sure, but certainly it was never intended to allow
lesser <= split <= greater.
But you might be right, perhaps we have two bugs to fix here.

[peterbell10]

You might be correct that the sliding midpoint is broken as well though, but I have never seen it break.

Assuming this is an issue, the chances of it adversely effecting the result of a query is astronomically small. You would need to have the query radius exactly overlap with the point on the split line. In which case, the result is borderline anyway and you would expect query results to be unstable just due to machine rounding.

[peterbell10]

So, this is more of a pedantically correct thing than a world-breaking bug.

[rainwoodman]

Is it more conventional to use r'\n'?

[rainwoodman]

What about adding a query test here? Is the new balanced tree build covered by a test?

[peterbell10]

I've fixed the un-compacted tree to use strict less-than, which means this tree always generates exactly 3 leaves. So, I can just assert that all the indices from the ones are in the first leaf and all the twoes in the second.

[sturlamolden]

is this good to merge now? I think it is.

[peterbell10]

Yes, this should be ready to go.

[tylerjereddy]

Looks like the kdtree experts are +1 to merge, CI is green, and I don't see anything too suspicious from a look through myself.

[tylerjereddy]

thanks @peterbell10 @rainwoodman @sturlamolden

[KelSolaar]

Thanks guys!