我正在尝试使用'pd.read_csv'读取一个1.3 GB的csv文件，其中包含两列和19,333行到Python的pandas数据帧中，但是它一直生成错误消息，说'CParserError：错误标记数据。 C错误：内存不足'，我尝试了许多在线发布的建议，比如使用'chunksize'，但它似乎不起作用，只产生'内核死了，重启'。这是运行'pd.read_csv'时的输出。
import pandas as pd
import numpy as np
import os
os.chdir("/home/swhan/Downloads")
CORPUS = pd.read_csv('10k_2005_2008_file.csv')
Traceback (most recent call last):
File "<ipython-input-1-8136c4f0354a>", line 7, in <module>
CORPUS = pd.read_csv('10k_2005_2008_file.csv')
File "/home/swhan/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 646, in parser_f
return _read(filepath_or_buffer, kwds)
File "/home/swhan/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 401, in _read
data = parser.read()
File "/home/swhan/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 939, in read
ret = self._engine.read(nrows)
File "/home/swhan/anaconda2/lib/python2.7/site-packages/pandas/io/parsers.py", line 1508, in read
data = self._reader.read(nrows)
File "pandas/parser.pyx", line 848, in pandas.parser.TextReader.read (pandas/parser.c:10415)
File "pandas/parser.pyx", line 870, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:10691)
File "pandas/parser.pyx", line 924, in pandas.parser.TextReader._read_rows (pandas/parser.c:11437)
File "pandas/parser.pyx", line 911, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:11308)
File "pandas/parser.pyx", line 2024, in pandas.parser.raise_parser_error (pandas/parser.c:27037)
CParserError: Error tokenizing data. C error: out of memory
实际上，csv文件由两列组成，一列用于ID，另一列用于每个ID的长文本信息，其中一个子集如下所示：
id text
12 python pandas read data of the form ...
13 how to remove file does not exist error ...
41 pandas unable to find files ...
99 issue with python is not a simple problem ...
有没有办法将这个文件读入pandas的dataframe对象？顺便说一句，我的桌面有32GB RAM。提前谢谢！
使用'chunksize'
替代尝试使用Python代码
df = pd.DataFrame()
reader = pd.read_csv("10k_2005_2008_file.csv", chunksize=10**3)
for chunk in reader:
df = pd.concat([df, chunk], ignore_index=True)
df
Out[6]:
ID text
0 255618 ['ITEM1.BUSINESSIn this annual report onForm10...
1 94740 ['Item 1. Business.GeneralCommunity CapitalCor...
2 145200 ['ITEM 1.BUSINESSGeneralCommunityBank Shares o...
3 145201 ['ITEM 1. BUSINESSGeneralCommunity Bank Share...
4 145202 ['Item 1. BusinessGeneralCommunity Bank Shares...
5 145203 ['Item1.BusinessGeneralCommunityBank Shares of...
6 221548 ['Item1.BusinessOverviewTravelzoo Inc. (the Co...
7 121633 ['Item1. BusinessGeneralSterling Financial Cor...
8 172796 ['Item 1. BusinessGeneralWe are a Maryland cor...
9 172797 ['Item 1. BusinessGeneralWe are a Maryland cor...
10 121632 ['Item 1.BusinessGeneralCompanyGrowthProfitabi...
11 28995 ['ITEM 1. Business.(Dollars in millions)We res...
12 28994 ['ITEM 1. Business.GeneralAt December31, 2004,...
13 28997 ['Item1.Business.GeneralService Corporation In...
14 28996 ['ITEM 1. Business.GeneralAt December31, 2004,...
15 118636 ['Item1.BusinessWe are a broadcast company pri...
16 28993 ['ITEM 1. Business.GeneralAt December31, 2004,...
17 101760 ['ITEM1.BUSINESSCorporateProfileCognex Corpora...
18 145752 ['Item 1: Election of Directors; Nomineesfor D...
19 94744 ['ITEM1.BUSINESS.GeneralCommunityCapital Corpo...
20 28999 ['Item1.Business.GeneralService Corporation In...
21 28998 ['Item1.Business.GeneralService Corporation In...
22 1868 ['ITEM1.BUSINESSCompany OverviewWe are a world...
23 269745 ['Item1"BusinessThe CompanyThe 2004 Reorganiza...
24 181343 ['ITEM 1. BUSINESSMKS Instruments, Inc. ("the...
25 220768 ['ITEM1. BUSINESS General The Company Sierr...
26 181345 ['Item1.BusinessMKS Instruments, Inc. (the Com...
27 145750 ['Item1. Business BurlingtonNorthern Santa F...
28 181346 ['Item1.BusinessMKS Instruments, Inc. (the Com...
29 145751 ['Item 1: Election of Directors; Nominees for ...
... ...
19303 26477 ['ITEM1.BUSINESS Precision Castparts Corp. (P...
19304 256145 ['Item1 Business,Item1A Risk Factors, and Item...
19305 222814 ['Item1. Business. General Our company, Rock...
19306 73641 ['ITEM 1. BUSINESSGENERALTexas Regional Bancsh...
19307 66997 ['ITEM 1. BUSINESSOur CompanyWe are a leading ...
19308 66996 ['ITEM 1. BUSINESSOur CompanyWe are a leading ...
19309 66994 ['ITEM1. BUSINESS Our Company We are a leadi...
19310 66993 ['ITEM 1. BUSINESS Our CompanyWe are a leadi...
19311 7929 ['Item1. Business(a)General development of bus...
19312 114251 ['Item1.BusinessGeneralTerra Nitrogen Company,...
19313 114250 ['Item1 BusinessGeneralTerra Nitrogen Company,...
19314 198077 ['Item1. BusinessGeneral DescriptionTeam Finan...
19315 162197 ["ITEM 1. BUSINESSWintrust Financial Corporati...
19316 25524 ['Item 1. BusinessEnvironmental. Contamination...
19317 190015 ['Item 1. Description of Business.GeneralEVCI ...
19318 5634 ['Item 1.BusinessGeneral CDI Corp. (the Compa...
19319 5635 ['Item 1.BusinessGeneral CDI Corp. (the Compa...
19320 190932 ['ITEM 1. BUSINESSORGANIZATION AND GENERAL B...
19321 190933 ['ITEM 1. BUSINESSORGANIZATION AND GENERAL B...
19322 5632 ['Item 1.BusinessGeneral CDI Corp., (the Comp...
19323 5633 ['Item 1.BusinessGeneral CDI Corp. (the Compa...
19324 38349 ['Item 1. BusinessThe CompanyNatures SunshineP...
19325 222816 ['Item1 above.Weoperate on a 52/53 week fiscal...
19326 222815 ['Item1. Business.GeneralOur company, Rockwell...
19327 213793 ['Item1.BusinessTvia,Inc. is a fabless semicon...
19328 8489 ['ITEM1.BusinessCrown Crafts, Inc. (the Compan...
19329 224247 ['Item1.Business GENERAL We are asolutions...
19330 198076 ['Item1. BusinessGeneral DescriptionTeam Finan...
19331 34149 ['Item1. BusinessVF Corporation, organized in ...
19332 34148 ['Item1 in PartI, Items 5, 6, 7, 7A, 8 and 9A ...
[19333 rows x 2 columns]
答案 0 :(得分：1)
注意但值得注意的是，concat（因此追加）会生成数据的完整副本，并且不断重复使用 此功能可以创造显着的性能。如果你需要 使用多个数据集上的操作，使用列表推导。
frames = [ process_your_file(f) for f in files ]
result = pd.concat(frames)
所以尝试这种方法：
reader = pd.read_csv("10k_2005_2008_file.csv", chunksize=10**3)
df = pd.concat([x for x in reader], ignore_index=True)