Question

我正在使用pandas / dask进行计算，我将数据存储在磁盘上的镶木地板文件中。问题是，我有一个专栏＆＃39; time＆＃39;还有一个叫做时间的索引。我想保留两者。当我存储数据然后再加载它时，我收到以下错误：

import pyarrow as pa
import pyarrow.parquet as pq
%matplotlib inline
dfx.to_dict()

Out[115]: 
{'close': {Timestamp('2017-06-30 01:31:00'): 154.99958999999998,
  Timestamp('2017-06-30 01:32:00'): 154.99958999999998,
  Timestamp('2017-06-30 01:33:00'): 154.01109,
  Timestamp('2017-06-30 01:34:00'): 154.01109,
  Timestamp('2017-06-30 01:35:00'): 152.60051000000001},
 'time': {Timestamp('2017-06-30 01:31:00'): Timestamp('2017-06-30 01:31:00'),
  Timestamp('2017-06-30 01:32:00'): Timestamp('2017-06-30 01:32:00'),
  Timestamp('2017-06-30 01:33:00'): Timestamp('2017-06-30 01:33:00'),
  Timestamp('2017-06-30 01:34:00'): Timestamp('2017-06-30 01:34:00'),
  Timestamp('2017-06-30 01:35:00'): Timestamp('2017-06-30 01:35:00')}}

# set index column 
dfx.set_index('time', drop=False, inplace=True)

dfx.head()
Out[117]: 
                                   time      close
time                                              
2017-06-30 01:31:00 2017-06-30 01:31:00  154.99959
2017-06-30 01:32:00 2017-06-30 01:32:00  154.99959
2017-06-30 01:33:00 2017-06-30 01:33:00  154.01109
2017-06-30 01:34:00 2017-06-30 01:34:00  154.01109
2017-06-30 01:35:00 2017-06-30 01:35:00  152.60051

# store to parquet file format
tdfx = pa.Table.from_pandas(dfx)
pq.write_table(tdfx, 'data.parquet' )


# recovering
dfx = pq.read_table('data.parquet').to_pandas()
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-119-5e9d7cd2ea0d> in <module>()
      1 # recovering
----> 2 dfx = pq.read_table('data.parquet').to_pandas()

pyarrow/table.pxi in pyarrow.lib.Table.to_pandas (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:37990)()

/home/ghildebrand/anaconda3/envs/p36/lib/python3.6/site-packages/pyarrow/pandas_compat.py in table_to_blockmanager(options, table, memory_pool, nthreads)
    296         i = schema.get_field_index(name)
    297         if i != -1:
--> 298             col = table.column(i)
    299             index_name = (None if is_unnamed_index_level(name)
    300                           else name)

pyarrow/table.pxi in pyarrow.lib.Table.column (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:38622)()

IndexError: Table column index 2 is out of range

这是pyarrow中的一个错误，或者这是不可能的镶木地板或我正在做其他错误？

更新：删除冗余列＆＃34; time＆＃34;并保持只有索引解决。所以我想这个问题是在镶木地板中创建了一组独特的列标识符集。

Answer 1

对我来说有点儿马车。我打开了一个错误报告https://issues.apache.org/jira/browse/ARROW-1754，让我们继续在那里讨论。

加载镶木地板文件时pyarrow的问题，其中索引具有冗余列

1 个答案: