Question

我正在尝试使用pandas.csv_read读取57MB的文件。该文件包含一个标题（5行），后缀整数值和结束浮点值：

info         
       2681087         53329       1287215       1287215         53328
RSA                    53328         53328       1287215             0
(I14)           (I14)           (d25.15)            (d25.15)            
F                          1         5332   
           1
          33
          61
          92
         128
         ...
         165
         205
         239
         272
    0.112474585277959E+09
    0.126110931411177E+09
    0.515995872032845E+09
    0.126110931411175E+09
   -0.194634413074014E+09
    0.112474585277950E+09
    ...

当我读取txt文件时：

import pandas as pd
pd.read_csv(file, skiprows=5+n_int_values, header=None, engine='c', 
            dtype=np.float, low_memory=False)

结果是错误：

---------------------------------------------------------------------------
CParserError                              Traceback (most recent call last)
<ipython-input-118-699921ac7a12> in <module>()
----> 1 a=pd.read_csv(loc, skiprows=5+n_coloums+n_rows, header=None, engine='c', low_memory=False, error_bad_lines=False)

C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in parser_f(filepath_or_buffer, sep, dialect, compression, doublequote, escapechar, quotechar, quoting, skipinitialspace, lineterminator, header, index_col, names, prefix, skiprows, skipfooter, skip_footer, na_values, na_fvalues, true_values, false_values, delimiter, converters, dtype, usecols, engine, delim_whitespace, as_recarray, na_filter, compact_ints, use_unsigned, low_memory, buffer_lines, warn_bad_lines, error_bad_lines, keep_default_na, thousands, comment, decimal, parse_dates, keep_date_col, dayfirst, date_parser, memory_map, float_precision, nrows, iterator, chunksize, verbose, encoding, squeeze, mangle_dupe_cols, tupleize_cols, infer_datetime_format, skip_blank_lines)
    468                     skip_blank_lines=skip_blank_lines)
    469 
--> 470         return _read(filepath_or_buffer, kwds)
    471 
    472     parser_f.__name__ = name

C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in _read(filepath_or_buffer, kwds)
    254         return parser
    255 
--> 256     return parser.read()
    257 
    258 _parser_defaults = {

C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in read(self, nrows)
    713                 raise ValueError('skip_footer not supported for iteration')
    714 
--> 715         ret = self._engine.read(nrows)
    716 
    717         if self.options.get('as_recarray'):

C:\Anaconda\lib\site-packages\pandas\io\parsers.pyc in read(self, nrows)
   1162 
   1163         try:
-> 1164             data = self._reader.read(nrows)
   1165         except StopIteration:
   1166             if nrows is None:

pandas\parser.pyx in pandas.parser.TextReader.read (pandas\parser.c:7426)()

pandas\parser.pyx in pandas.parser.TextReader._read_rows (pandas\parser.c:8377)()

pandas\parser.pyx in pandas.parser.raise_parser_error (pandas\parser.c:20728)()

CParserError: Error tokenizing data. C error: Buffer overflow caught - possible malformed input file.

任何想法为什么以及如何解决这个问题。

注意：如果我使用engine='python'，则会正常加载txt文件。

Note2 ：error_bad_lines=False不会改变任何内容。

更新这发生在pandas 0.16.0上。在较旧的版本 - 0.14.1。它工作正常。

更新2 该错误已在pandas 0.16.1

中修复

Answer 1

该错误已在熊猫0.16.1中修复

pandas read_csv错误在Windows上

1 个答案: