Question

我希望通过网络抓取Google搜索结果，并希望获得显示的第一批信息。如何指定从中提取文本的特定HTML路径？

import requests
import lxml
from bs4 import BeautifulSoup

city = "Potomac"
suffix = "Weather"
query = city + " " + suffix

url = "https://www.google.com/search?q=" + query

# Now have the best URL for a city
results = requests.get(url)

# Extract all content
src = results.content

# Get HTML soup of all content on that page
soup = BeautifulSoup(src, "lxml")
# print(soup.prettify())

# Try to find and print specific places
precip = soup.findAll("span", attrs = {"id": "wob_pp"})

我原本希望找到所有的span标签（这是我要提取的数据的标签），但是许多嵌套的span标签不会出现。

Answer 1

您应该通过HTTP标头进行请求。 HTTP标头允许客户端和服务器将附加信息与请求或响应一起传递。

results = requests.get(url, headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36'})

如何从Beautiful Soup中的嵌套标签中提取文本？

1 个答案: