如何从MongoDB中获取数据?

时间:2017-11-22 21:12:52

标签: python mongodb

我正试图将一个mongo数据库中的网址写入我的scrapy蜘蛛和 然后用Selenium迭代它。

这段代码一切顺利,我只想迭代“urls”我已经刮掉并储存在mongodb中。我怎么能这样做?

这是我的代码:

from selenium import webdriver
from time import sleep

driver = webdriver.Chrome('/path')
driver.get('https://www.website.com')

#login page
email = driver.find_element_by_name('email')
email.send_keys('email@gmail.com')
sleep(0.5)

password  = driver.find_element_by_name('pass')
password.send_keys('mypass')
sleep(0.5)

sign_in_button = driver.find_element_by_xpath('//*[@type="button"]')
sign_in_button.click()
sleep(5)

#pre-home
next_button = driver.find_element_by_class_name('_2pii')
next_button.click()
sleep(5)

#pages to scrape
urls = ('https://www.website.com/page1','https://www.website.com/page2')

for url in urls:
    driver.get(url)
    sleep(5)

    try:
        add_button = driver.find_element_by_xpath('//*[contains(concat( " ", @class, " " ), concat( " ", "_5caa", " " ))]')
        add_button.click()

    except:
        pass

0 个答案:

没有答案