Question

将蜘蛛部署到Scraping Cloud。它会收集文件的下载链接，并将这些文件保存在Google Cloud存储桶中。它在本地运行时有效。但是，当部署到Scraping Hub时，它将返回以下错误：

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 95, in crawl
    six.reraise(*exc_info)
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 77, in crawl
    self.engine = self._create_engine()
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 102, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 70, in __init__
    self.scraper = Scraper(crawler)
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/scraper.py", line 71, in __init__
    self.itemproc = itemproc_cls.from_crawler(crawler)
  File "/usr/local/lib/python2.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/usr/local/lib/python2.7/site-packages/scrapy/middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
    mod = import_module(module)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/app/__main__.egg/tceq_crawler/pipelines.py", line 4, in <module>
ImportError: cannot import name GCSFilesStore

它在本地运行，并且能够将文件上传到GCloud存储桶。有一个仅为蜘蛛创建的服务帐户。我导出了JSON凭证文件，并将其直接粘贴到凭证对象中。

在YML文件中，我指定了需求文件。在requirements.txt文件中，我指定了所有必需的软件包。

在这一点上，我不确定为什么在导入FilesPipeline很好时为什么不能在Scrapy Cloud上导入GCSFilesStore。

这是我的代码：

pipelines.py

# -*- coding: utf-8 -*-

from scrapy.pipelines.files import FilesPipeline
from scrapy.pipelines.files import GCSFilesStore
from scrapy.http.request import Request

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html

class TceqCrawlerPipeline(object):
    def process_item(self, item, spider):
        return item

class CustomFileNamePipeline(FilesPipeline):

    def get_media_requests(self, item, info):
        return [Request(x, meta={'file_name': item["file_name"]})
                for x in item.get('file_urls', [])]

    def file_path(self, request, response=None, info=None):
        return request.meta['file_name']

class GCSFilesStoreJSON(GCSFilesStore):
    CREDENTIALS = {
        "type": "service_account",
        "project_id": "project_id",
        "private_key_id": "FROM JSON FILE",
        "private_key": "-----BEGIN PRIVATE KEY-----FROM JSON FILE-----END PRIVATE KEY-----\n",
        "client_email": "FROM JSON FILE",
        "client_id": "FROM JSON FILE",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/FROM JSON FILE"
    }
    def __init__(self, uri):
        from google.cloud import storage
        client = storage.Client.from_service_account_info(self.CREDENTIALS)
        bucket, prefix = uri[5:].split('/', 1)
        self.bucket = client.bucket(bucket)
        self.prefix = prefix

class GCSFilePipeline(FilesPipeline):
    def __init__(self, store_uri, download_func=None, settings=None):
        super(GCSFilePipeline, self).__init__(store_uri,download_func,settings)

settings.py

DOWNLOADER_MIDDLEWARES = {'scrapy_crawlera.CrawleraMiddleware': 300}
CRAWLERA_ENABLED = True
CRAWLERA_APIKEY = 'API_KEY'


ITEM_PIPELINES = {
    'tceq_crawler.pipelines.GCSFilePipeline': 1,
    'tceq_crawler.pipelines.CustomFileNamePipeline': 200
}
##GOOGLE CLOUD STORAGE
FILES_STORE = 'gs://bucket-name/'
GCS_PROJECT_ID = 'project_id'

scrapinghub.yml

project: 12345

requirements:
  file: requirements.txt

requirements.txt

pyasn1==0.4.1
google-api-core==1.8.1                     
google-auth==1.6.3                     
google-cloud-core==0.29.1                    
google-cloud-storage==1.14.0                  
google-resumable-media==0.3.2                    
googleapis-common-protos==1.5.8

Answer 1

您需要将scrapy dependency添加到您的requirements.txt：

scrapy=1.6.0
pyasn1==0.4.1
google-api-core==1.8.1                     
google-auth==1.6.3                     
google-cloud-core==0.29.1                    
google-cloud-storage==1.14.0                  
google-resumable-media==0.3.2                    
googleapis-common-protos==1.5.8

在Scrapy Cloud上获取Spider以使用GCSFilesStore在Google Cloud Storage上存储文件并获取ImportError

1 个答案: