我正试图将一个mongo数据库中的网址写入我的scrapy蜘蛛和 然后用Selenium迭代它。
这段代码一切顺利,我只想迭代“urls”我已经刮掉并储存在mongodb中。我怎么能这样做?
这是我的代码:
from selenium import webdriver
from time import sleep
driver = webdriver.Chrome('/path')
driver.get('https://www.website.com')
#login page
email = driver.find_element_by_name('email')
email.send_keys('email@gmail.com')
sleep(0.5)
password = driver.find_element_by_name('pass')
password.send_keys('mypass')
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//*[@type="button"]')
sign_in_button.click()
sleep(5)
#pre-home
next_button = driver.find_element_by_class_name('_2pii')
next_button.click()
sleep(5)
#pages to scrape
urls = ('https://www.website.com/page1','https://www.website.com/page2')
for url in urls:
driver.get(url)
sleep(5)
try:
add_button = driver.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "_5caa", " " ))]')
add_button.click()
except:
pass