使用pandas在csv文件中读取时出现的编码问题

时间:2015-03-25 22:12:30

标签: python pandas character-encoding

我正在尝试读取包含一些不规则行的大型csv。我一直在收到错误,说明“期待113个字符,遇到114个字符”。我已经尝试丢弃大于预期分隔字符数的行,但后来遇到了另一个错误。代码和错误如下所示:

import pandas as pd
import csv

f_name = 'some_file.txt'
    with open(f_name,'r') as file:
    lines = file.readlines()

bad_lines= []
for i,line in enumerate(lines):
    if line.count('\t') != 112:
        bad_lines.append(i)
df = pd.read_csv(f_name,header=0,sep='\t',skiprows=bad_lines,error_bad_lines=False)
df.head()

这会产生以下警告和错误:

Skipping line 3332366: expected 113 fields, saw 168

Skipping line 3829308: expected 113 fields, saw 114

Skipping line 4819090: expected 113 fields, saw 114

Skipping line 4832856: expected 113 fields, saw 114

Skipping line 5170065: expected 113 fields, saw 114

Skipping line 5198811: expected 113 fields, saw 168

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-19-ea2fca7ce11a> in <module>()
      1 df = pd.read_csv(f_name,header=0,sep='\t',skiprows=bad_lines,error_bad_lines=False)
----> 2 df.head()

/Library/Python/2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
    236                 self.write_format_data(format_dict, md_dict)
    237                 self.log_output(format_dict)
--> 238             self.finish_displayhook()
    239 
    240     def cull_cache(self):

/Library/Python/2.7/site-packages/IPython/kernel/zmq/displayhook.pyc in finish_displayhook(self)
     70         sys.stderr.flush()
     71         if self.msg['content']['data']:
---> 72             self.session.send(self.pub_socket, self.msg, ident=self.topic)
     73         self.msg = None
     74 

/Library/Python/2.7/site-packages/IPython/kernel/zmq/session.pyc in send(self, stream, msg_or_type, content, parent, ident, buffers, track, header, metadata)
    647         if self.adapt_version:
    648             msg = adapt(msg, self.adapt_version)
--> 649         to_send = self.serialize(msg, ident)
    650         to_send.extend(buffers)
    651         longest = max([ len(s) for s in to_send ])

/Library/Python/2.7/site-packages/IPython/kernel/zmq/session.pyc in serialize(self, msg, ident)
    551             content = self.none
    552         elif isinstance(content, dict):
--> 553             content = self.pack(content)
    554         elif isinstance(content, bytes):
    555             # content is already packed, as in a relayed message

/Library/Python/2.7/site-packages/IPython/kernel/zmq/session.pyc in <lambda>(obj)
     83 # disallow nan, because it's not actually valid JSON
     84 json_packer = lambda obj: jsonapi.dumps(obj, default=date_default,
---> 85     ensure_ascii=False, allow_nan=False,
     86 )
     87 json_unpacker = lambda s: jsonapi.loads(s)

/Library/Python/2.7/site-packages/zmq/utils/jsonapi.pyc in dumps(o, **kwargs)
     38         kwargs['separators'] = (',', ':')
     39 
---> 40     s = jsonmod.dumps(o, **kwargs)
     41 
     42     if isinstance(s, unicode):

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.pyc in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, encoding, default, sort_keys, **kw)
    248         check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    249         separators=separators, encoding=encoding, default=default,
--> 250         sort_keys=sort_keys, **kw).encode(obj)
    251 
    252 

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.pyc in encode(self, o)
    208         if not isinstance(chunks, (list, tuple)):
    209             chunks = list(chunks)
--> 210         return ''.join(chunks)
    211 
    212     def iterencode(self, o, _one_shot=False):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 4: ordinal not in range(128)

我本来期望我的预防措施已经删除了这些内容,但我也不完全了解如何解决编码问题。奇怪的角色完全有可能进入数据,但我没有处理编码问题的经验,而且我无法适应我在网上找到的任何其他解决方案。

提前感谢您的帮助!

0 个答案:

没有答案