Question

我在Windows上使用Google App Engine devserver 1.3.5和Python 2.5.4时出现了一个奇怪的错误。

CSV中的示例行：

EQS,550,foobar,"<some><html><garbage /></html></some>",odp,Ti4=,http://url.com,success

错误：

..................................................................................................................[ERROR   ] [Thread-1] WorkerThread:
Traceback (most recent call last):
  File "C:\Program Files\Google\google_appengine\google\appengine\tools\adaptive_thread_pool.py", line 150, in WorkOnItems
    status, instruction = item.PerformWork(self.__thread_pool)
  File "C:\Program Files\Google\google_appengine\google\appengine\tools\bulkloader.py", line 695, in PerformWork
    transfer_time = self._TransferItem(thread_pool)
  File "C:\Program Files\Google\google_appengine\google\appengine\tools\bulkloader.py", line 852, in _TransferItem
    self.request_manager.PostEntities(self.content)
  File "C:\Program Files\Google\google_appengine\google\appengine\tools\bulkloader.py", line 1296, in PostEntities
    datastore.Put(entities)
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.py", line 282, in Put
    req.entity_list().extend([e._ToPb() for e in entities])
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore.py", line 687, in _ToPb
    properties = datastore_types.ToPropertyPb(name, values)
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_types.py", line 1499, in ToPropertyPb
    pbvalue = pack_prop(name, v, pb.mutable_value())
  File "C:\Program Files\Google\google_appengine\google\appengine\api\datastore_types.py", line 1322, in PackString
    pbvalue.set_stringvalue(unicode(value).encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 36: ordinal not in range(128)
[INFO    ] Unexpected thread death: Thread-1
[INFO    ] An error occurred. Shutting down...
..[ERROR   ] Error in Thread-1: 'ascii' codec can't decode byte 0xe8 in position 36: ordinal not in range(128)

错误是由base64字符串的问题产生的，每行中都有一个字符串？

KGxwMAoobHAxCihTJ0JJT0VFJwpwMgpJMjYxMAp0cDMKYWEu

KGxwMAoobHAxCihTJ01BVEgnCnAyCkkyOTQwCnRwMwphYS4=

数据加载器：

class CourseLoader(bulkloader.Loader):
    def __init__(self):
        bulkloader.Loader.__init__(self, 'Course',
                                   [('dept_code', str),
                                    ('number', int),
                                    ('title', str),
                                    ('full_description', str),
                                    ('unparsed_pre_reqs', str),
                                    ('pickled_pre_reqs', lambda x: base64.b64decode(x)),
                                    ('course_catalog_url', str),
                                    ('parse_succeeded', lambda x: x == 'success')
                                   ])

loaders = [CourseLoader]

有没有办法告诉回溯哪一行导致错误？

更新：看起来有两个字符导致错误：è和®。如何让Google App Engine处理它们？

Answer 1

看起来CSV的某些行有一些非ascii数据（可能是LATIN SMALL LETTER E WITH GRAVE - 这就是{-1}}在ISO-8859-1中的含义，然而你却是将其映射到0xe8（应为str，我相信CSV应该在utf-8中。）

要查找文本文件的任何行是否包含非ascii数据，一个简单的Python代码段将有所帮助，例如：

unicode

Google App Engine：批量数据上传时出现UnicodeDecode错误

1 个答案: