Question

我的情况是我使用setting.py

将图像保存到亚马逊s3

AWS_ACCESS_KEY_ID = "xxxxxx"
AWS_SECRET_ACCESS_KEY = "xxxxxx"
IMAGES_STORE = "s3://bucketname/image_store/"

我遇到一个问题，即终端中的scrapy信息没有显示错误信息，
但图像没有下载到amazon s3服务器。

有没有人遇到类似的问题，scrapy在下载图像丢失时没有显示错误信息？

没有错误输出，所以我不知道在哪里解决问题

这是我的终端控制台，如：

2014-09-13 05:04:08+0000 [scrapy] INFO: Enabled item pipelines: ImagesPipeline, ShPipeline
2014-09-13 05:04:08+0000 [email5] INFO: Spider opened
2014-09-13 05:04:08+0000 [email5] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-09-13 05:04:08+0000 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
....
2014-09-13 05:04:08+0000 [email5] DEBUG: Scraped from <200 http://www.test.com/test/test>
{'image_urls': [u'http://xxx/.jpg'],
 'images': [{'checksum': 'xxx',
             'path': 'full/xxx.jpg',
             'url': 'http://xxx/.jpg'}],
 'link': 'http://www.test.com/',
  }
2014-09-13 05:04:08+0000 [email5] INFO: Closing spider (finished)
2014-09-13 05:04:08+0000 [email5] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 621,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 71214,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'file_count': 1,
 'file_status_count/downloaded': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2014, 9, 12, 1, 33, 33, 611656),
 'item_scraped_count': 1,
 'log_count/DEBUG': 6,
 'log_count/INFO': 7,
 'response_received_count': 2,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2014, 9, 12, 1, 33, 31, 766911)}
2014-09-12 01:33:33+0000 [email4] INFO: Spider closed (finished)

更新我的问题：

我发现有时终端没有显示错误。有时会出现这样的错误：

2014-09-13 05:04:08+0000 [-] ERROR: Unhandled error in Deferred:
2014-09-13 05:04:08+0000 [-] Unhandled Error
.....
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>

scrapy图像没有下载，也没有错误信息

0 个答案: