ENH: Make categorical repr nicer.

[jseabold]

Make looking at Categorical types a little nicer. Needs some tests still, but works fine locally so far.

[jseabold]

Added tests.

[jreback]

@jseabold this is nice......can you hook up travis? (prob just need to flip the switch), then

git checkin --amend -C HEAD to recommit last commit

[jseabold]

Should be going now. Ping me if it's not. Maybe needed a setup lag.

[jreback]

does not appear to have taken.....

[jseabold]

Appears to be going now.

[jseabold]

Well, I don't see the banner here, but I see it running on travis. I dunno.

[cpcloud]

travis usually takes a couple of minutes to actually start, that's how it rolls

[cpcloud]

i see the banner

[jseabold]

Anyone know off the top of their head the 2.6 errors? I assume it's a unicode comparison issue...

[cpcloud]

that kind of looks like a bug

python 2.6:

Categorical:
[a, b, b, a, a, c, c, c]
Levels (3): Index([a, b, c], dtype=object)

In [10]: factor.levels
Out[10]: Index([u'a', u'b', u'c'], dtype=object)

python 2.7

In [1]: factor = Categorical.from_array(['a','b','b','a','a','c','c','c'])

In [2]: factor
Out[2]:
Categorical:
[a, b, b, a, a, c, c, c]
Levels (3): Index(['a', 'b', 'c'], dtype=object)

In [3]: factor.levels
Out[3]: Index([u'a', u'b', u'c'], dtype=object)

[cpcloud]

or possibly a difference in numpy since categorical repr calls np.array_repr...my python 2.6 has np 1.6.1 and py 2.7 up there has 1.7.1

[jreback]

yep...this should be com.pprint_thing(self.levels)

[jreback]

or maybe self.levels.format()

[cpcloud]

self.levels.format() does the correct thing

[jseabold]

I'm almost inclined to just fix the tests here and live with the numpy inconsistency unless there's another way around this. pprint_thing drops the object name and I like knowing levels is an Index. E.g.,

>>> np.array_repr(np.arange(3))
'array([0, 1, 2])'

>>> com.pprint_thing(np.arange(3))
u'[0, 1, 2]'

[cpcloud]

what's wrong with self.levels.format()?

[jseabold]

It's a list (for 1.8.x)? So either array_repr fails with an error or I'm back to pprint_thing. Either way I lose the Index([...]) information.

[jseabold]

Or of course, leaving it, I lose the Index info. I could just do something like "Index(%s)" but I hoped to avoid this in case levels is ever not an Index for any reason, but I guess the tests will catch a change like that now.

[cpcloud]

oh duh sorry yes you're right.

[jseabold]

In a delicious bit of irony I'm the one that changed the unicode repr of numpy object arrays.

numpy/numpy#459

[cpcloud]

yep np.array2string is different in those two versions of numpy

[jseabold]

Thanks. For the record, y'all beat my attempts to build Python 2.6, the whole numpy stack, and pandas in the background of my work.

[cpcloud]

@jseabold building the stack is not fun. i get strange issues with matplotlib and tkinter so i can't do any plotting...i'm also accumulating a bit of enmity for the timedelta64 (non-)functionality of numpy 1.6 (it silently doesn't and then sometimes does convert units). i would suggest avoiding numpy 1.6 like the plague if you have the option. python 2.6 is probably ok

[jreback]

hah....we have had fair share of issues with py2.6 lately!

here's a fun bug in python (that was actually not fixed): http://bugs.python.org/issue2325

[jseabold]

Yes, I switched to openblas recently and my hacked together numpy/distutils is not working for scipy (numpy tests pass) for 2.6.

[jseabold]

Fixed the tests.

[cpcloud]

gets bit unwieldy for a large number of levels (i tried 100), but my opinion here doesn't matter that much i never use Categorical

[jseabold]

I left a TODO in there for this. It's admittedly not handled, but I can't imagine having categorical variables with too many categories. The degrees of freedom loss is too prohibitive in estimation. Maybe it could be useful in some machine learning contexts, but I don't know how ML people use factors in R. I suspect they don't.

[cpcloud]

totally fine by me. like i said, i never found a need for this

[jreback]

@jseabold this looks ready? maybe squash a bit

@cpcloud ?

[cpcloud]

fine by me

[hayd]

Too late to throw out case the repr being valid python (so it can eval itself) :s ?

[jreback]

@jseabold this almost got lost...

can you rebase and we can merge it in....

thxs

[jseabold]

Rebased. There was a merge conflict, which I didn't check too closely, so make sure tests pass.

[jseabold]

Any idea what's going on here?

[jseabold]

Looks like a circular import introduced in 85f191c?

[jreback]

merged via 7c76086

thanks @jseabold