使用BeautifulSoup从Google搜索中抓取网址

Question

我的代码是

import urllib
from bs4 import BeautifulSoup
import requests
import webbrowser

text = 'LastName, FirstName'
text = urllib.parse.quote_plus(text)

url = 'https://google.com/search?q=' + text

response = requests.get(url)

我想获取第一个搜索结果的网址，我该怎么做？

Answer 1

您可能要考虑执行此任务：

import urllib
from bs4 import BeautifulSoup
from selenium import webdriver


text = 'LastName, FirstName'
text = urllib.parse.quote_plus(text)

url = 'https://google.com/search?q=' + text

driver = webdriver.Firefox()
driver.get(url)

soup = BeautifulSoup(driver.page_source)
results_links = soup.find('div',{'class':'srg'}).find_all("a")
print(results_links[0].get('href'))

输出：

  

https://www.quora.com/What-is-meant-by-first-name-and-last-name