ENH: tolerance for Float64Index including join / reindex-nearest

[hsuominen]

When trying to intersect two Index objects containing floats I get the following unexpected behavior:

>>> new_index = pd.Index(np.arange(0.0,1.0,0.1),dtype='float64')
>>> new_index2 = pd.Index(np.arange(0.5,1.0,0.1),dtype='float64')
>>> intersection = new_index.intersection(new_index2)
Float64Index([0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], dtype='float64')
Float64Index([0.5, 0.6, 0.7, 0.8, 0.9], dtype='float64')
Float64Index([0.5], dtype='float64')

Where I would expect the intersection to equal index2.

Pandas version string below:

INSTALLED VERSIONS

commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-45-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.16.0
nose: 1.3.4
Cython: 0.21.2
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: 2.4.1
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.2
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None

[dsm054]

This is basically suggesting we introduce a float tolerance for alignment, because your two values aren't the same:

>>> repr(new_index.values[6])
'0.60000000000000009'
>>> repr(new_index2.values[1])
'0.59999999999999998'

[shoyer]

The problem here is unfortunately rather inherent in the nature of floating point numbers. In general, producing the same set of floating point numbers two different ways will produce numbers that are not exactly equal.

We could do more in pandas to handle floating point tolerance automatically, though the exact implementation remains to be worked out and there are some potential performance issues. See here for more discussion: #9530

[jreback]

Here's what getting called
as these are both monotonic. Would need a tol as @shoyer and @dsm054 point out for these types of comparisons.

Not a bad idea, but would require a bit of effort.

[jreback]

You can do this to in-effect get what you want. This would need a tolerance as well (in the reindexer) so make it robust.

In [28]: target, indexer = new_index.reindex(new_index2,method='nearest')

In [29]: target
Out[29]: Float64Index([0.5, 0.6, 0.7, 0.8, 0.9], dtype='float64')

In [30]: indexer
Out[30]: array([5, 6, 7, 8, 9])