我尝试用 Scrapy 刮板做一个项目，到目前为止效果很好，从零开始。要使用 Scrapy，我需要有一个“win32api”库。我已将其安装在我的虚拟环境中，但它无法识别并不断向我显示错误。
(scr_env) C:\Users\antoi\Documents\Programming\Tests\bass2\scraper>python -m pip install win32api
Collecting win32api
Could not find a version that satisfies the requirement win32api (from versions: )
No matching distribution found for win32api
You are using pip version 10.0.1, however version 20.3.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
(scr_env) C:\Users\antoi\Documents\Programming\Tests\bass2\scraper>scrapy crawl sephora -o perfumes.json
2021-01-18 10:45:56 [scrapy.utils.log] INFO: Scrapy 2.4.1 started (bot: nosetime_scraper)
2021-01-18 10:45:56 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 18.9.0, Python 3.6.7 (v3.6.7
:6ec5cf24b7, Oct 20 2018, 13:35:33) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1h 22 Sep 2020), cryptography 3.2.1, Platform Windows-10-10.0.19041-S
P0
2021-01-18 10:45:56 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2021-01-18 10:45:56 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'nosetime_scraper',
'CONCURRENT_REQUESTS': 1,
'COOKIES_ENABLED': False,
'DOWNLOAD_DELAY': 7,
'NEWSPIDER_MODULE': 'nosetime_scraper.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['nosetime_scraper.spiders'],
'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) '
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 '
'Safari/537.36'}
2021-01-18 10:45:56 [scrapy.extensions.telnet] INFO: Telnet Password: 16ea497e99400f83
2021-01-18 10:45:56 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2021-01-18 10:45:57 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "http"
Traceback (most recent call last):
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 49, in _load_handler
dhcls = load_object(path)
File "c:\python36\lib\site-packages\scrapy\utils\misc.py", line 62, in load_object
mod = import_module(module)
File "c:\python36\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 2, in <module>
from scrapy.core.downloader.handlers.http11 import (
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 13, in <module>
from twisted.internet.endpoints import TCP4ClientEndpoint
File "c:\python36\lib\site-packages\twisted\internet\endpoints.py", line 41, in <module>
from twisted.internet.stdio import StandardIO, PipeAddress
File "c:\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
from twisted.internet import _win32stdio
File "c:\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in <module>
import win32api
ModuleNotFoundError: No module named 'win32api'
2021-01-18 10:45:57 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.http.HTTPDownloadHandler" for scheme "https"
Traceback (most recent call last):
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 49, in _load_handler
dhcls = load_object(path)
File "c:\python36\lib\site-packages\scrapy\utils\misc.py", line 62, in load_object
mod = import_module(module)
File "c:\python36\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 2, in <module>
from scrapy.core.downloader.handlers.http11 import (
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 13, in <module>
from twisted.internet.endpoints import TCP4ClientEndpoint
File "c:\python36\lib\site-packages\twisted\internet\endpoints.py", line 41, in <module>
from twisted.internet.stdio import StandardIO, PipeAddress
File "c:\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
from twisted.internet import _win32stdio
File "c:\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in <module>
import win32api
ModuleNotFoundError: No module named 'win32api'
2021-01-18 10:45:57 [scrapy.core.downloader.handlers] ERROR: Loading "scrapy.core.downloader.handlers.s3.S3DownloadHandler" for scheme "s3"
Traceback (most recent call last):
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 49, in _load_handler
dhcls = load_object(path)
File "c:\python36\lib\site-packages\scrapy\utils\misc.py", line 62, in load_object
mod = import_module(module)
File "c:\python36\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\s3.py", line 3, in <module>
from scrapy.core.downloader.handlers.http import HTTPDownloadHandler
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\http.py", line 2, in <module>
from scrapy.core.downloader.handlers.http11 import (
File "c:\python36\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 13, in <module>
from twisted.internet.endpoints import TCP4ClientEndpoint
File "c:\python36\lib\site-packages\twisted\internet\endpoints.py", line 41, in <module>
from twisted.internet.stdio import StandardIO, PipeAddress
File "c:\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
from twisted.internet import _win32stdio
File "c:\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in <module>
import win32api
ModuleNotFoundError: No module named 'win32api'
Unhandled error in Deferred:
2021-01-18 10:45:57 [twisted] CRITICAL: Unhandled error in Deferred:
Traceback (most recent call last):
File "c:\python36\lib\site-packages\scrapy\crawler.py", line 192, in crawl
return self._crawl(crawler, *args, **kwargs)
File "c:\python36\lib\site-packages\scrapy\crawler.py", line 196, in _crawl
d = crawler.crawl(*args, **kwargs)
File "c:\python36\lib\site-packages\twisted\internet\defer.py", line 1613, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "c:\python36\lib\site-packages\twisted\internet\defer.py", line 1529, in _cancellableInlineCallbacks
_inlineCallbacks(None, g, status)
--- <exception caught here> ---
File "c:\python36\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "c:\python36\lib\site-packages\scrapy\crawler.py", line 87, in crawl
self.engine = self._create_engine()
File "c:\python36\lib\site-packages\scrapy\crawler.py", line 101, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "c:\python36\lib\site-packages\scrapy\core\engine.py", line 69, in __init__
self.downloader = downloader_cls(crawler)
File "c:\python36\lib\site-packages\scrapy\core\downloader\__init__.py", line 83, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "c:\python36\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "c:\python36\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "c:\python36\lib\site-packages\scrapy\utils\misc.py", line 62, in load_object
mod = import_module(module)
File "c:\python36\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\python36\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 24, in <module>
from twisted.web.client import ResponseFailed
File "c:\python36\lib\site-packages\twisted\web\client.py", line 41, in <module>
from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
File "c:\python36\lib\site-packages\twisted\internet\endpoints.py", line 41, in <module>
from twisted.internet.stdio import StandardIO, PipeAddress
File "c:\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
from twisted.internet import _win32stdio
File "c:\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in <module>
import win32api
builtins.ModuleNotFoundError: No module named 'win32api'
2021-01-18 10:45:57 [twisted] CRITICAL:
Traceback (most recent call last):
File "c:\python36\lib\site-packages\twisted\internet\defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "c:\python36\lib\site-packages\scrapy\crawler.py", line 87, in crawl
self.engine = self._create_engine()
File "c:\python36\lib\site-packages\scrapy\crawler.py", line 101, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "c:\python36\lib\site-packages\scrapy\core\engine.py", line 69, in __init__
self.downloader = downloader_cls(crawler)
File "c:\python36\lib\site-packages\scrapy\core\downloader\__init__.py", line 83, in __init__
self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "c:\python36\lib\site-packages\scrapy\middleware.py", line 53, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "c:\python36\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "c:\python36\lib\site-packages\scrapy\utils\misc.py", line 62, in load_object
mod = import_module(module)
File "c:\python36\lib\importlib\__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\python36\lib\site-packages\scrapy\downloadermiddlewares\retry.py", line 24, in <module>
from twisted.web.client import ResponseFailed
File "c:\python36\lib\site-packages\twisted\web\client.py", line 41, in <module>
from twisted.internet.endpoints import HostnameEndpoint, wrapClientTLS
File "c:\python36\lib\site-packages\twisted\internet\endpoints.py", line 41, in <module>
from twisted.internet.stdio import StandardIO, PipeAddress
File "c:\python36\lib\site-packages\twisted\internet\stdio.py", line 30, in <module>
from twisted.internet import _win32stdio
File "c:\python36\lib\site-packages\twisted\internet\_win32stdio.py", line 9, in <module>
import win32api
ModuleNotFoundError: No module named 'win32api'
而且我确实安装了 pywin32：
(scr_env) C:\Users\antoi\Documents\Programming\Tests\bass2\scraper>python -m pip install pypiwin32
Requirement already satisfied: pypiwin32 in c:\users\antoi\documents\programming\tests\bass2\scraper\scr_env\lib\site-packages (223)
Requirement already satisfied: pywin32>=223 in c:\users\antoi\documents\programming\tests\bass2\scraper\scr_env\lib\site-packages (from pypiwin32) (300)
You are using pip version 10.0.1, however version 20.3.3 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
并得到同样的错误。
我也试过XXX提出的方案
pip install -i https://pypi.douban.com/simple pypiwin32
它已安装，但我仍有 ModuleNotFoundError。
我还尝试了在我之前的 Scrapy 抓取工具中似乎有效的方法：
python -m pip install pywin32-ctypes