Question

我在OSX El Capitan 10.11.2上的Python 2.7.10上使用Pandas 0.18.1，如果我没有设置engine='python'，我在使用read_csv()读取UTF-16文件时遇到问题。

文档说明Python解析器功能更完整，因此Pandas可能会默认使用C解析器，并且它还不支持UTF-16。有人可以确认是否是这种情况，或者这里是否还有其他事情发生？

以下是最小化的再现方案：

alanwagner : ~ ∴ pip2.7 freeze | grep pandas
pandas==0.18.1
alanwagner : ~ ∴ cat test.csv 
col1,col2
val1,val2
alanwagner : ~ ∴ python
Python 2.7.10 (default, Oct 23 2015, 18:05:06) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.read_csv('test.csv', encoding='utf8').to_csv('test-utf16.csv', encoding='utf16', index=False)
>>> 
alanwagner : ~ ∴ cat test-utf16.csv 
??col1,col2
val1,val2
alanwagner : ~ ∴ python
Python 2.7.10 (default, Oct 23 2015, 18:05:06) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.read_csv('test-utf16.csv', encoding='utf16')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/pandas/io/parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Library/Python/2.7/site-packages/pandas/io/parsers.py", line 315, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Library/Python/2.7/site-packages/pandas/io/parsers.py", line 645, in __init__
    self._make_engine(self.engine)
  File "/Library/Python/2.7/site-packages/pandas/io/parsers.py", line 799, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/Library/Python/2.7/site-packages/pandas/io/parsers.py", line 1213, in __init__
    self._reader = _parser.TextReader(src, **kwds)
  File "pandas/parser.pyx", line 520, in pandas.parser.TextReader.__cinit__ (pandas/parser.c:5129)
  File "pandas/parser.pyx", line 701, in pandas.parser.TextReader._get_header (pandas/parser.c:7665)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_16.py", line 16, in decode
    return codecs.utf_16_decode(input, errors, True)
UnicodeDecodeError: 'utf16' codec can't decode byte 0x63 in position 2: truncated data
>>> pd.read_csv('test-utf16.csv', encoding='utf16', engine='python')
   col1  col2
0  val1  val2
>>>

我可以通过将文件从UTF-16转换为UTF-8，然后再将其加载到Pandas DataFrame中来解决这个问题。

Answer 1

是的，这是真的！您可以改为-

pd.read_csv('test-utf16.csv', encoding='utf-16')

我还没有找到原因，但是您应该在不设置引擎的情况下阅读它。

Python Pandas read_csv（encoding ='utf16'）仅适用于engine ='python'？

1 个答案: