Question

当我尝试使用pandas.Timestamp作为索引编写镶木地板文件时，无法读回。出现关于时间戳转换的错误。

pytz.exceptions.UnknownTimeZoneError: '-5:00'

我已经针对pyarrow 0.11.1运行了此代码，并且运行良好，当切换到0.15.0时会中断。我认为这是pyarrow处理日期时间对象的方式，而是通过pytz解析时区。

此代码适用于pyarrow 0.11.1，中断适用于0.15.0

import pandas as pd
import pyarrow.parquet as pq
import pyarrow as pa
import json

myjson = ['{"received_time":"2019-10-13T00:00:09.915-05:00","mydata":1.0}',
          '{"received_time":"2019-10-13T00:00:10.915-05:00","mydata":1.0}',
          '{"received_time":"2019-10-13T00:00:11.915-05:00","mydata":1.0}']
iterator = 0
q=[]
for i in myjson:
    q.append(json.loads(i))
    iterator = iterator + 1
df = pd.DataFrame.from_records(q)
tindex = pd.to_datetime(df['received_time'],format='%Y-%m-%dT%H:%M:%S.%f%z')
df['received_time'] = tindex
df.set_index('received_time',inplace=True)
tbl = pa.Table.from_pandas(df)
pq.write_table(tbl,'tmp.pq')

readdf = pd.read_parquet('tmp.pq')

我希望它可以正常工作，而不会出现错误，

Traceback (most recent call last):
  File "<input>", line 21, in <module>
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/io/parquet.py", line 296, in read_parquet
    return impl.read(path, columns=columns, **kwargs)
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/io/parquet.py", line 125, in read
    path, columns=columns, **kwargs
  File "pyarrow/array.pxi", line 468, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 1238, in pyarrow.lib.Table._to_pandas
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 697, in table_to_blockmanager
    all_columns)
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 778, in _reconstruct_index
    table, result_table, descr, field_name_to_metadata)
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pyarrow/pandas_compat.py", line 835, in _extract_index_level
    .dt.tz_convert(col.type.tz))
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/accessor.py", line 93, in f
    return self._delegate_method(name, *args, **kwargs)
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/indexes/accessors.py", line 109, in _delegate_method
    result = method(*args, **kwargs)
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/accessor.py", line 93, in f
    return self._delegate_method(name, *args, **kwargs)
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/indexes/datetimelike.py", line 813, in _delegate_method
    result = operator.methodcaller(name, *args, **kwargs)(self._data)
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pandas/core/arrays/datetimes.py", line 955, in tz_convert
    tz = timezones.maybe_get_tz(tz)
  File "pandas/_libs/tslibs/timezones.pyx", line 84, in pandas._libs.tslibs.timezones.maybe_get_tz
  File "pandas/_libs/tslibs/timezones.pyx", line 99, in pandas._libs.tslibs.timezones.maybe_get_tz
  File "/home/<redacted>/PycharmProjects/data_analysis/venv/lib64/python3.6/site-packages/pytz/__init__.py", line 181, in timezone
    raise UnknownTimeZoneError(zone)
pytz.exceptions.UnknownTimeZoneError: '-05:00'

这是pandas.show_versions()

的输出

pd.show_versions()
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 3.10.0-862.el7.x86_64
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8
pandas           : 0.25.1
numpy            : 1.15.4
pytz             : 2019.3
dateutil         : 2.7.5
pip              : 10.0.1
setuptools       : 39.1.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.0.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.15.0
pytables         : None
s3fs             : None
scipy            : 1.3.0
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

我不确定这是不是pyarrow问题还是大熊猫问题，但是我确实知道pyarrow切换到了这样处理日期时间：

    if isinstance(col.type, pa.lib.TimestampType):
        index_level = (pd.Series(values).dt.tz_localize('utc')
                       .dt.tz_convert(col.type.tz))

会调用pytz库，而之前没有。我不是正确构造了镶木地板文件（在这种情况下，写入时会出现解析错误），或者是大熊猫或镶木地板库如何处理时区。

有什么想法吗？

无法使用熊猫时间戳记写入有效的实木复合地板文件

0 个答案: