Scrapyd S3 Feed导出“Peer连接重置”

时间:2014-11-25 16:19:29

标签: amazon-s3 scrapy boto scrapyd

我正在运行Scrapyd并将FEED_URI设置为导出到S3,但是在我刮掉的最后我收到了以下错误。请注意,当scrape开始时,它已成功将几百kb的数据上传到存储桶,然后在最后抛出此错误:

2014-11-24 10:11:23+0000 [word] ERROR: Error storing csv feed (2285242 items) in: s3://kitchen.bucket/FoodItem.csv Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap self.__bootstrap_inner() File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker result = context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/usr/local/lib/python2.7/dist-packages/scrapy/contrib/feedexport.py", line 101, in _store_in_thread key.set_contents_from_file(file) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1291, in set_contents_from_file chunked_transfer=chunked_transfer, size=size) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 748, in send_file chunked_transfer=chunked_transfer, size=size) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 949, in _send_file_internal query_args=query_args File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request retry_handler=retry_handler File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1068, in make_request retry_handler=retry_handler) File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 939, in _mexe request.body, request.headers) File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 842, in sender http_conn.send(chunk) File "/usr/lib/python2.7/httplib.py", line 805, in send self.sock.sendall(data) File "/usr/lib/python2.7/ssl.py", line 329, in sendall v = self.send(data[count:]) File "/usr/lib/python2.7/ssl.py", line 298, in send v = self._sslobj.write(data) socket.error: [Errno 104] Connection reset by peer

boto issue 2207类似。我使用gbirke&#39; MultiFeedExporter,并在我的两件商品上收到了类似的错误。

0 个答案:

没有答案