Question

我写了这段代码登录我的FB帐户，并使用Selenuim和BeautifulSoup获取页面上的所有grouplinks，但BeautifulSoup用法无效。

我想知道如何在相同的代码中使用Selenuim和BeautifulSoup。

我不想使用Facebook API;我想使用Selenium和BeautifulSoup。

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import httplib2
from BeautifulSoup import BeautifulSoup, SoupStrainer


usr = raw_input('--> ')
pwd = raw_input('--> ')
poo = raw_input('--> ')

driver = webdriver.Firefox()
# or you can use Chrome(executable_path="/usr/bin/chromedriver")
driver.get("https://www.facebook.com/groups/?category=membership")
assert "Facebook" in driver.title
elem = driver.find_element_by_id("email")
elem.send_keys(usr)
elem = driver.find_element_by_id("pass")
elem.send_keys(pwd)
elem.send_keys(Keys.RETURN)

scheight = .1
while scheight < 9.9:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight/%s);" % scheight)
    scheight += .01
soup = BeautifulSoup(html)
http = httplib2.Http()
status, response = ('https://www.facebook.com/groups/?category=membership')

count = 0
for link in BeautifulSoup(response, parseOnlyThese=SoupStrainer('a')):
     count = count + 1
print 'Count: ', count 

for tag in BeautifulSoup(('a')):
    if link.has_key('href'):
        if '/groups/' in link['href']:

          print link['href']


elem = driver.find_element_by_css_selector(".input.textInput")
elem.send_keys(poo)
elem = driver.find_element_by_css_selector(".selected")
elem.send_keys(Keys.RETURN)
elem.click()
time.sleep(5)

Answer 1

您从未声明html。

Selenium的webdriver有一个page_source方法可以使用：

soup = BeautifulSoup(driver.page_source)

第二次错误更新

你的行，

status, response = ('https://www.facebook.com/groups/?category=membership')

正在尝试将该字符串分配给status和response。没有分配给response，因此该变量未定义。

Answer 2

我认为BeautifulSoup没有回复链接吗？

我在BeautifulSoup中找到

的链接

soup = BeautifulSoup( html )
for i in soup.find_all( 'a' ):
if '/groups/' in i.get( 'href' ):
    print( i.get( 'href' ) )

使用Beautifulsoup和Selenium从具有特定单词的网页获取链接

2 个答案: