ENH: merge multi-index with a multi-index

[jreback]

#3662 is about merging a single-level index with a mi

This is is about a multi-multi merge

# example described there:
In [11]: df = pd.DataFrame(np.random.randn(10,5))
    ...: df1 = df[range(0, 3)].set_index([0, 2])
    ...: df2 = df[range(2, 5)].set_index([2, 4])
    ...: res = df1.join(df2, how='inner')  # empty. Do I need to specify something here?
    ...: exp = pd.DataFrame({1: df[1], 3: df[3]})
    ...: exp.index = df[2]

[jreback]

Here's a partial implementation: jreback@0c38215

[PKEuS]

I tried to get your partial implementation working. As-is, it complained about non-fitting shape. I enabled the commented code then, which I rewrote to this one:
PKEuS@90c307c

The commented code gave these tuples:

[('2', 'gb00b03mlx29', '233'), ('3', 'gb00b03mlx29', '234'), ('3', 'lu0197800237
', '235'), (4.0, nan, '180')]

It now creates this tuples

[('2', 'asset_id', '233'), ('2', 'asset_id', '234'), ('2', 'asset_id', '235'), (
'3', 'asset_id', '233'), ('3', 'asset_id', '234'), ('3', 'asset_id', '235'), ('3
', 'asset_id', '180'), ('3', 'asset_id', '181')]
['household_id', 'asset_id', 't']

Compared to the tuples created by your commented code, it looks correct to me. (You can ignore the line with asset_id=nan for now)

However, it crashes after returning the MultiIndex somewhere in the merging code called after merging the index. This crashs the python interpreter, which makes nosetests abort without printing any results.

Any ideas?

[jreback]

the tuples here are not that complicated
it's creating the correct indexers that is tricky
these are the indexers that allow a take_nd from the original to the new ones
essentially it is a mapping that is of the new shape with values that correspond to the original shape
that are possibly duplicated

[PKEuS]

Indeed. The reason for the crash were non-fitting sizes of indexers and indexes. Fixed that. Now it creates two objects which are imho identical, but assert_frame_equal returns false. And surprisingly, it creates the same output for "inner" and "outer" merge. However, I would like to have one of these working first.

If I print the two dataframes (result and expected), they look equal:

                               share  log_return
household_id asset_id     t
2            gb00b03mlx29 233   0.60    0.096050
                          234   0.60   -0.065241
                          235   0.60    0.035324
3            gb00b03mlx29 233   0.15    0.096050
                          234   0.15   -0.065241
                          235   0.15    0.035324
             lu0197800237 180   0.60    0.030254
                          181   0.60    0.036997

[8 rows x 2 columns]
                               share  log_return
household_id asset_id     t
2            gb00b03mlx29 233   0.60    0.096050
                          234   0.60   -0.065241
                          235   0.60    0.035324
3            gb00b03mlx29 233   0.15    0.096050
                          234   0.15   -0.065241
                          235   0.15    0.035324
             lu0197800237 180   0.60    0.030254
                          181   0.60    0.036997

[8 rows x 2 columns]

However, assert_frame_equals prints:

======================================================================
FAIL: test_join_multi_levels2 (pandas.tools.tests.test_merge.TestMergeMulti)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python\pandas\pandas\tools\tests\test_merge.py", line 1121, in test_j
oin_multi_levels2
    assert_frame_equal(result,expected)
  File "C:\Python\pandas\pandas\util\testing.py", line 523, in assert_frame_equa
l
    assert_index_equal(left.index, right.index)
  File "C:\Python\pandas\pandas\util\testing.py", line 467, in assert_index_equa
l
    right.dtype))
nose.proxy.AssertionError: AssertionError: [index] left [object household_id  as
set_id      t
2             gb00b03mlx29  233
                            234
                            235
3             gb00b03mlx29  233
                            234
                            235
              lu0197800237  180
                            181], right [household_id  asset_id      t
2             gb00b03mlx29  233
                            234
                            235
3             gb00b03mlx29  233
                            234
                            235
              lu0197800237  180
                            181 object]

[jreback]

hmm...grab the 2 indexes before the assertion and look at them, they should both be multi-indexes with the same names / levels and such, try left.equals(right). If that works, then its the test that is not comparing these correctly; if it fails then its possible your created index is someone not being created quite equal

[PKEuS]

The indexes look equal, however, both

print(result.index.equals(expected.index))

and

print(result.equals(expected))

print "False".

Besides from that, I feel I reinvented the wheel somehow, since my code hardcodes an "inner merge". Since achieving an "outer merge" or left/right is harder that way, I guess I should use some already existing method. Those seem to not work with multiindex objects, even if join levels are given. Has anybody an idea about the best way to use the existing mehtods on the multiindex?

[PKEuS]

At the moment I have a problem (which is the reason for the test failures) with this line:

t.extend(tuple(np.array(left).take(left_indexer)))

It creates an array from left (which is needed to use the take method), but all non-string indices become strings due to that conversion. Do you have any suggestions on how to solve that?