Question

我尝试部署通过Portia创建的蜘蛛。 Portia，scrapyd - 都是最新版本。

我只是通过命令运行 scrapyd 服务器：

scrapyd

我在本地计算机上获得此结果：

$> cd PROJECT_PATH_HERE;
$> scrapyd-deploy
Packing version 1463420166
Deploying to project "retail" in http://X.X.X.X:6800/addversion.json
Server response (200):
{"status": "error", "message": "AttributeError: 'list' object has no attribute 'iteritems'", "node_name": "vaua0048313.online-vm.com"}

在服务器端（ Ubuntu 12.04.5 LTS ）：

root@server:~# scrapyd
2016-05-16 17:35:42+0000 [-] Log opened.
2016-05-16 17:35:42+0000 [-] twistd 16.1.1 (/usr/bin/python 2.7.3) starting up.
2016-05-16 17:35:42+0000 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2016-05-16 17:35:42+0000 [-] Site starting on 6800
2016-05-16 17:35:42+0000 [-] Starting factory <twisted.web.server.Site instance at 0x2f5d950>
2016-05-16 17:35:42+0000 [Launcher] Scrapyd 1.1.0 started: max_proc=12, runner='scrapyd.runner'
2016-05-16 17:36:06+0000 [_GenericHTTPChannelProtocol,0,78.111.181.96] Unhandled Error
        Traceback (most recent call last):
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 1767, in allContentReceived
            req.requestReceived(command, path, version)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/http.py", line 768, in requestReceived
            self.process()
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 183, in process
            self.render(resrc)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/server.py", line 234, in render
            body = resrc.render(self)
        --- <exception caught here> ---
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/webservice.py", line 17, in render
            return JsonResource.render(self, txrequest)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/utils.py", line 19, in render
            r = resource.Resource.render(self, txrequest)
          File "/usr/local/lib/python2.7/dist-packages/twisted/web/resource.py", line 250, in render
            return m(request)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/webservice.py", line 79, in render_POST
            spiders = get_spider_list(project, version=version)
          File "/usr/local/lib/python2.7/dist-packages/scrapyd/utils.py", line 116, in get_spider_list
            raise RuntimeError(msg.splitlines()[-1])
        exceptions.RuntimeError: AttributeError: 'list' object has no attribute 'iteritems'

2016-05-16 17:36:06+0000 [-] "78.111.181.96" - - [16/May/2016:17:36:05 +0000] "POST /addversion.json HTTP/1.1" 200 135 "-" "Python-urllib/2.7"

我做错了什么？

Answer 1

我认为问题来自你的setting.py。当您使用scrapy版本1.10.0时，ITEM_PIPELINES应由下面的dict格式指定，

ITEM_PIPELINES = {
    'coolscrapy.pipelines.ArticleDataBasePipeline': 5,
}

Answer 2

将 Portia_Projects / spiders / setting.py 中的 ITEM_PIPELINES 修改为

ITEM_PIPELINES = {'slybot.dupefilter.DupeFilterPipeline':1}

无法将scrapy部署到scrapyd服务器

2 个答案: