Question

我有大量的文件和解析器。我要做的是剥离所有非utf-8符号并将数据放入mongodb。目前我有这样的代码。

with open(fname, "r") as fp:
    for line in fp:
        line = line.strip()
        line = line.decode('utf-8', 'ignore')
        line = line.encode('utf-8', 'ignore')

不知怎的，我仍然收到错误

bson.errors.InvalidStringData: strings in documents must be valid UTF-8: 
1/b62010montecassianomcir\xe2\x86\x90ta0\xe2\x86\x90008923304320733/290066010401040101506055soccorin

我不明白。有一些简单的方法吗？

UPD ：似乎Python和Mongo不同意Utf-8有效字符串的定义。

Answer 1

尝试以下代码行而不是最后两行。希望它有所帮助：

line=line.decode('utf-8','ignore').encode("utf-8")

Answer 2

对于python 3，如本主题的评论中所述，您可以这样做：

line = bytes(line, 'utf-8').decode('utf-8', 'ignore')

＆＃39;忽略＆＃39;参数可以防止在无法解码任何字符时引发错误。

如果您的行已经是字节对象（例如b'my string'），那么您只需要使用decode('utf-8', 'ignore')对其进行解码。

Answer 3

不处理utf-8字符的示例

import string

test=u"\n\n\n\n\n\n\n\n\n\n\n\n\n\nHi <<First Name>>\nthis is filler text \xa325 more filler.\nadditilnal filler.\n\nyet more\xa0still more\xa0filler.\n\n\xa0\n\n\n\n\nmore\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nfiller.\x03\n\t\t\t\t\t\t    almost there \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nthe end\n\n\n\n\n\n\n\n\n\n\n\n\n"

print ''.join(x for x in test if x in string.printable)

Answer 4

with open(fname, "r") as fp:
for line in fp:
    line = line.strip()
    line = line.decode('cp1252').encode('utf-8')

从字符串中删除每个非utf-8符号

4 个答案: