我在Scrapy项目中使用Selenium,并且我的代码在我的计算机上可以正常工作。当我尝试在AWS上运行相同版本的代码时,当Scrapy尝试为Firefox设置Selenium Webdriver时,它会说:
2018-11-01 05:27:38 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method SeleniumMiddleware.spider_opened of <navigator.middlewares.SeleniumMiddleware object at 0x7ffae89bc470>>
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/twisted/internet/defer.py", line 149, in maybeDeferred
result = f(*args, **kw)
File "/usr/local/lib/python3.5/dist-packages/pydispatch/robustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "/home/ubuntu/search/decapod/navigator/navigator/middlewares.py", line 213, in spider_opened
self.driver = webdriver.Firefox(executable_path = path, firefox_options=options)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
keep_alive=True)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: newSession
这是我启动Selenium的代码:
if not self.uses_selenium:
spider.log("SELENIUM NOT ENABLED", logging.WARNING)
return
spider.log("SELENIUM ENABLED", logging.WARNING)
path = os.getcwd() + "/geckodriver"
options = webdriver.FirefoxOptions()
if self.headless:
options.add_argument('--headless')
options.add_argument("--window-size=1600,900")
options.add_argument('--disk-cache=true')
options.add_argument('--load-images=false')
options.add_argument("--user-agent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0")
self.driver = webdriver.Firefox(executable_path = path, firefox_options=options)
我想念一些包裹吗?