Question

我正在尝试使用python scrapy工具从bitcointalk.org网站中提取有关用户及其在论坛上发布的公钥的信息。

我在网上找到了这段代码，对它进行了更改，以便它在我想要的网站上运行，但我遇到了错误，AttributeError响应对象没有属性文本。

以下是参考代码

class BitcointalkSpider(CrawlSpider):
name = "bitcointalk"
allowed_domains = ["bitcointalk.org"]

start_urls = ["https://bitcointalk.org/index.php"]

rules = (
    Rule(SgmlLinkExtractor(deny=[
        'https://bitcointalk\.org/index\.php\?action=ignore',
        'https://bitcointalk\.org/index\.php\?action=profile',
        ], 
        allow_domains='bitcointalk.org'), callback='parse_item', follow=True),
)

def parse_item(self, response):
    sel = Selector(response)
    sites = sel.xpath('//tr[contains(@class, "td_headerandpost")]')
    items = []
    for site in sites:
        item = BitcoinItem()
        item["membername"] = site.xpath('.//td[@class="poster_info"]/b/a/text()').extract()
        addresses = site.xpath('.//div[contains(@class, "signature")]/text()').re(r'(1[1-9A-HJ-NP-Za-km-z]{26,33})')
        if item["membername"] and addresses:
            addr_list = set()
            for addr in addresses:
                if (bcv.check_bc(addr)):
                    addr_list.add(addr)
            item["address"] = addr_list
            if len(addr_list) > 0:
                items.append(item)
    return items

我收到的错误是：

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line     102, in iter_errback
yield next(it)
File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py", line 22, in <genexpr>
return (_set_referer(r) for r in result or ())
File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
return (r for r in result or () if _filter(r))
File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
return (r for r in result or () if _filter(r))
File "/usr/local/lib/python2.7/dist-packages/scrapy/spiders/crawl.py", line 72, in _parse_response
cb_res = callback(response, **cb_kwargs) or ()
File "/home/sunil/Desktop/Nikhil/Thesis/mit_bitcoin/bitcoin/spiders/bitcointalk_spider.py", line 24, in parse_item
sel = Selector(response)
File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/unified.py", line 63, in __init__
text = response.text
AttributeError: 'Response' object has no attribute 'text'

Answer 1

您的某个请求可能有问题，因为看起来至少有一个网址的响应您的抓取格式不正确。请求本身失败，或者您没有正确提出请求。

See here了解您的错误来源。

并see here了解您的请求可能格式不正确的原因。看起来Selector需要一个HtmlResponse对象或类似的类型。

属性错误响应对象没有属性＆＃39; text＆＃39;

1 个答案: