Question

我有一个在我的系统上运行的webscraper，我想将它迁移到PythonAnywhere，但是当我移动它现在它不起作用。

sendkeys似乎无法正常工作 - 执行以下代码后，我从未转到下一个网页，因此会触发属性错误。

我的代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import csv
import time



# Lists for functions
parcel_link =[]
token = []
csv_output = [ ]

# main scraping function
def getLinks(link):


# Open web browser and get url - 3 second time delay.

#Open web browser and get url - 3 second time delay.
driver.get(link)
time.sleep(3)

inputElement = driver.find_element_by_id("mSearchControl_mParcelID")
inputElement.send_keys(parcel_code+"*/n")
print("ENTER hit")

pageSource = driver.page_source
bsObj = BeautifulSoup(pageSource)
parcel_link.clear()
print(bsObj)
#pause = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.ID, "mResultscontrol_mGrid_RealDataGrid")))

for link in bsObj.find(id="mResultscontrol_mGrid_RealDataGrid").findAll('a'):
    parcel_link.append(link.text)
print(parcel_link)

for test in parcel_link:
    clickable = driver.find_element_by_link_text(test)
    clickable.click()
    time.sleep(2)

我尝试操作的链接是： https://ascendweb.jacksongov.org/ascend/%280yzb2gusuzb0kyvjwniv3255%29/search.aspx

我想发送：15-100 *

回溯：

03:12 ~/Tax_Scrape $ xvfb-run python3.4 Jackson_Parcel_script.py
Traceback (most recent call last):
  File "Jackson_Parcel_script.py", line 377, in <module>
    getLinks("https://ascendweb.jacksongov.org/ascend/%28biohwjq5iibvvkisd1kjmm45%29/result.aspx")
  File "Jackson_Parcel_script.py", line 29, in getLinks
    inputElement = driver.find_element_by_id("mSearchControl_mParcelID")
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 206, in find_element_by_id
    return self.find_element(by=By.ID, value=id_)
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 662, in find_element
    {'using': by, 'value': value})['value']
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 173, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: 'Unable to locate element: {"method":"id","selector":"mSearchControl_mParcelID"}' ; Stac
ktrace: 
    at FirefoxDriver.findElementInternal_ (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/driver_component.js:9470)
    at FirefoxDriver.findElement (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/driver_component.js:9479)
    at DelayedCommand.executeInternal_/h (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/command_processor.js:11455)
    at DelayedCommand.executeInternal_ (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/command_processor.js:11460)
    at DelayedCommand.execute/< (file:///tmp/tmpiuuqg3m7/extensions/fxdriver@googlecode.com/components/command_processor.js:11402) 
03:13 ~/Tax_Scrape $

Selenium Innitation：

for retry in range(3):
    try:
        driver = webdriver.Firefox()
        break
    except:
        time.sleep(3)

for parcel_code in token:
    getLinks("https://ascendweb.jacksongov.org/ascend/%28biohwjq5iibvvkisd1kjmm4 5%29/result.aspx")

PythonAnywhere使用FireFox的虚拟实例，假设像JSPhantom一样无头，所以我没有版本号。

任何帮助都会很棒

RS

Answer 1

好吧，也许PythonAnywhere上使用的浏览器无法足够快地加载网站。因此，不要time.sleep(3)尝试implicitly waiting元素。

隐式等待是告诉WebDriver对DOM进行轮询尝试查找一个或多个元素的时间量没有立即可用。默认设置为0.一旦设置，隐式等待是为WebDriver对象实例的生命周期设置的。

将time.sleep()与Selenium一起使用通常不是一个好主意。

并提供超过3秒，而implicitly_wait()指定等待元素的最长时间。
因此，如果您设置implicitly_wait(10)并加载页面，例如5秒，那么Selenium将仅等待5秒。

driver.implicitly_wait(10)

Selenium Code在本地工作，但不在PythonAnywhere上工作

1 个答案: