Question

从Scrapy结果中，标题中包含一个不需要的非ASCII字符代码\u2013（又名character(150)或en dash），例如u'Director/Senior Director \u2013 Pathology'。我正在尝试使用管道来移除常规\u2013 ,。但是下面的代码没有用。也不会报告任何错误消息。

from datetime import datetime
from hashlib import md5
from scrapy.exceptions import DropItem
from twisted.enterprise import adbapi
import re
import string

class ReplaceASC2InTitlePipeline(object):
"""replace unwanted ASCII characters in titles"""

ascii_to_filter = ["\u2013",]

def process_item(self, item, spider):
    for word in self.ascii_to_filter:
        desc = item.get('title')

        if (desc) and word in desc:
            spider.log("\u2013 in '%s' was replace" % (item['title']) )

            item['title']=item['title'].replace("\u2013", ",")
            return item
    else:
        return item

Answer 1

"\u2013"应该是unicode，所以只需替换：

ascii_to_filter = ["\u2013",]

使用：

ascii_to_filter = [u"\u2013",]

Answer 2

在阅读了这个stackoverflow帖子Replace non-ASCII characters...之后，我想出了这个代码，它将过滤掉标题中的所有非ASCII字符。对于我的情况，不需要非ASCII字符，因此它对我来说非常适合。

from datetime import datetime
from hashlib import md5
from scrapy.exceptions import DropItem
from twisted.enterprise import adbapi
import re
import string

class ReplaceASC2InTitlePipeline(object):
"""replace unwanted non-ASCII characters in titles"""

def process_item(self, item, spider):

    def remove_non_ascii(text):
        return ''.join(i for i in text if ord(i)<128)

    orig_titl = item.get('title')
    item['title'] = remove_non_ascii(orig_titl) 

    if item['title'] != orig_titl:
        spider.log("Non-ASCII character(s) was removed in '%s'" % (item['title']) )

    return item

Scrapy：使用管道替换不需要的非ASCII代码

2 个答案: