Question

在Python 3和熊猫中，在具有Windows 10和16 GB RAM的计算机上，我执行了以下命令以读取CSV文件：

import pandas as pd
socios = pd.read_csv('com_cpfs/socio.csv', converters={'cnpj': lambda x: str(x)})

“ socio.csv”文件具有1.69 GB的内容，大约有1600万行

出现此错误消息：

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-6-7b4f3c0b9089> in <module>
----> 1 socios = pd.read_csv('com_cpfs/socio.csv', converters={'cnpj': lambda x: str(x)})

c:\users\reinaldo\documents\code\receita\lib\site-packages\pandas\io\parsers.py in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision)
    676                     skip_blank_lines=skip_blank_lines)
    677 
--> 678         return _read(filepath_or_buffer, kwds)
    679 
    680     parser_f.__name__ = name

c:\users\reinaldo\documents\code\receita\lib\site-packages\pandas\io\parsers.py in _read(filepath_or_buffer, kwds)
    444 
    445     try:
--> 446         data = parser.read(nrows)
    447     finally:
    448         parser.close()

c:\users\reinaldo\documents\code\receita\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
   1034                 raise ValueError('skipfooter not supported for iteration')
   1035 
-> 1036         ret = self._engine.read(nrows)
   1037 
   1038         # May alter columns / col_dict

c:\users\reinaldo\documents\code\receita\lib\site-packages\pandas\io\parsers.py in read(self, nrows)
   1846     def read(self, nrows=None):
   1847         try:
-> 1848             data = self._reader.read(nrows)
   1849         except StopIteration:
   1850             if self._first_chunk:

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader.read()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._read_rows()

pandas\_libs\parsers.pyx in pandas._libs.parsers.TextReader._convert_column_data()

pandas\_libs\parsers.pyx in pandas._libs.parsers._maybe_upcast()

MemoryError:

请问，可以在熊猫中打开这么大的文件以创建数据框吗？

还是我需要其他方法？哪个？

如何在熊猫中打开具有数百万行的文件？

0 个答案: