Question

我使用python和BeautifulSoup库创建了一个脚本，以从网页中抓取某些内容。我感兴趣的内容位于该页面的What does that mean下。

更具体地说-我想解析的内容：

此标题下What does that mean下的所有内容，除了图片。

这是我到目前为止试图抓住的：

import requests
from bs4 import BeautifulSoup

link = "https://www.obd-codes.com/p0100"

def fetch_data(link):
    res = requests.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    [script.extract() for script in soup.select("script")]
    elem = [item.text for item in soup.select("h2:contains('What does that mean') ~ p")]
    print(elem)

if __name__ == '__main__':
    fetch_data(link)

但是，我尝试过的方法几乎可以提供该页面上我没有得到的所有内容。

如何从上一页获取What does that mean和What are some possible symptoms之间的内容？

PS，我不希望使用正则表达式。

Answer 1

您可以利用itertools.takewhile（official doc）函数来完成所需的操作：

import requests
from bs4 import BeautifulSoup

from itertools import takewhile

link = "https://www.obd-codes.com/p0100"

def fetch_data(link):
    res = requests.get(link)
    soup = BeautifulSoup(res.text,"lxml")
    [script.extract() for script in soup.select("script")]
    elems = [i.text for i in takewhile(lambda tag: tag.name != 'h2', soup.select("h2:contains('What does that mean') ~ *"))]
    print(elems)

if __name__ == '__main__':
    fetch_data(link)

打印：

['This diagnostic trouble code (DTC) is a generic powertrain code, which means that it applies to OBD-II equipped vehicles that have a mass airflow sensor. Brands include but are not limited to Toyota, Nissan, Vauxhall, Mercedes Benz, Mitsubishi, VW, Saturn, Ford, Jeep, Jaguar, Chevy, Infiniti, etc. Although generic, the specific repair steps may vary depending on make/model.', "The MAF (mass air flow) sensor is a sensor mounted in a vehicle's engine air intake tract downstream from the air filter, and is used to measure the volume and density of air being drawn into the engine. The MAF sensor itself only measures a portion of the air entering and that value is used to calculate the total volume and density of air being ingested.", '\n\n\n\n\xa0', '\n', 'The powertrain control module (PCM) uses that reading along with other sensor parameters to ensure proper fuel delivery at any given time for optimum power and fuel efficiency.', 'This P0100 diagnostic trouble code (DTC) means that there is a detected problem with the Mass Air Flow (MAF)\nsensor or circuit. The PCM detects that the actual MAF sensor frequency signal\nis not performing within the normal expected range of the calculated MAF value.', 'Note: Some MAF sensors also incorporate an air temperature sensor, which is another value used by the PCM for optimal engine operation.', 'Closely related MAF circuit trouble codes include:', '\nP0101 Mass or Volume Air Flow "A" Circuit Range/Performance\nP0102 Mass\nor Volume Air Flow "A" Circuit Low Input\nP0103 Mass\nor Volume Air Flow "A" Circuit High Input\nP0104 Mass or Volume Air Flow "A" Circuit Intermittent\n', 'Photo of a MAF sensor:']

编辑：

如果仅在<p>标记后只需要<h2>标记，请使用lambda tag: tag.name == 'p'。

Answer 2

还有另一种方法可以实现相同目的。让您的脚本继续运行，直到遇到标签h2。

import requests
from bs4 import BeautifulSoup

url = "https://www.obd-codes.com/p0100"

res = requests.get(url)
soup = BeautifulSoup(res.text,"lxml")
[script.extract() for script in soup.select("script")]
elem_start = [elem for elem in soup.select_one("h2:contains('What does that mean')").find_all_next()]
content = []
for item in elem_start:
    if item.name=='h2': break
    content.append(' '.join(item.text.split()))
print(content)

使用BeautifulSoup

2 个答案: