Skip to content

Hdf5/PyTables support for TimedeltaIndex #9635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JonasAbernot opened this issue Mar 12, 2015 · 2 comments
Closed

Hdf5/PyTables support for TimedeltaIndex #9635

JonasAbernot opened this issue Mar 12, 2015 · 2 comments
Labels
Bug Dtype Conversions Unexpected or buggy dtype conversions IO HDF5 read_hdf, HDFStore Timedelta Timedelta data type
Milestone

Comments

@JonasAbernot
Copy link
Contributor

When stored in HDFStore, TimedeltaIndex are converted into Int64Index.

df = pd.DataFrame(np.random.normal(size=(10,5)))
df.index = pd.timedelta_range(start='0s',periods=10,freq='1s')
store = pd.HDFStore('test.h5')
store['df'] = df
df = store['df']
print df.index
Int64Index([0, 1000000000, 2000000000, 3000000000, 4000000000, 5000000000, 6000000000, 7000000000, 8000000000, 9000000000], dtype='int64')

Although the backward conversion is easy (pd.TimedeltaIndex(df.index)) , it's surprising.

pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-46-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8

pandas: 0.15.2-269-ge3d1b0e
nose: 1.3.1
Cython: 0.20.1post0
numpy: 1.8.2
scipy: 0.13.3
statsmodels: None
IPython: 1.2.1
sphinx: 1.2.2
patsy: None
dateutil: 2.4.1
pytz: 2013b
bottleneck: None
tables: 3.1.1
numexpr: 2.2.2
matplotlib: 1.3.1
openpyxl: None
xlrd: None
xlwt: 0.7.5
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
httplib2: 0.8
apiclient: None
rpy2: None
sqlalchemy: 0.9.8
pymysql: None
psycopg2: None

I would have been happy to help and add this myself, but I really have no idea were is this hiding. I don't even know if I should have looked toward PyTables instead of here.

Thanks !

@jreback
Copy link
Contributor

jreback commented Mar 12, 2015

This works just fine when a TimedeltaIndex is a column. This is missing a converted in _convert_index/_unconvert_index . Pull requests are welcome

In [36]: df.reset_index().to_hdf('test3.h5','df',mode='w')

In [37]: pd.read_hdf('test3.h5','df')
Out[37]: 
     index         0         1         2         3         4
0 00:00:00 -0.238141  0.416589  1.309751 -1.869346  1.050291
1 00:00:01 -0.972722 -0.722563  1.358120  1.905053  0.340453
2 00:00:02 -0.239058  0.847760 -1.783979  0.249137 -0.197160
3 00:00:03  0.084814 -0.749522 -1.289472  0.786044 -0.520832
4 00:00:04  0.846480  0.116762  3.107453 -1.161420 -1.043352
5 00:00:05 -1.912248  1.010266 -2.267783 -1.191424  1.371217
6 00:00:06 -0.020378 -0.332079  2.224500 -0.476001  0.021783
7 00:00:07 -0.349609 -0.357767 -0.360366 -2.099308 -0.229146
8 00:00:08  0.139164 -2.290294 -0.102519  1.554782  0.291216
9 00:00:09 -0.959462  0.028526  0.555051 -0.852721  0.019570

@jreback jreback added this to the 0.16.1 milestone Mar 12, 2015
@jreback jreback added Bug Dtype Conversions Unexpected or buggy dtype conversions IO HDF5 read_hdf, HDFStore Timedelta Timedelta data type labels Mar 12, 2015
JonasAbernot added a commit to JonasAbernot/pandas that referenced this issue Apr 3, 2015
@jreback
Copy link
Contributor

jreback commented Apr 4, 2015

closed by #9672

@jreback jreback closed this as completed Apr 4, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Dtype Conversions Unexpected or buggy dtype conversions IO HDF5 read_hdf, HDFStore Timedelta Timedelta data type
Projects
None yet
2 participants