Question

我正在尝试从重复了以下html几次的网站上抓取一些信息：

<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup</h2>
   <span class="reference-number">REF NO. A1400.5</span>

我正在尝试刮擦产品名称和文本“ REF NO。A1400.5”。我需要在同一页面上抓取多个产品名称和参考编号，并将它们存储在列表中。我尝试过：

product_new = []
product_ref = []

for caption in soup.find_all(class_='product-details'):
    product_name_new = caption.find(class_='product-name').text
    product_new.append(product_name_new)
    product_name_ref = (soup.select_one("span[class*=reference]").text)
    product_ref.append(product_name_ref)    
product_size_new = len(product_new)
print("Setup Complete", product_size_new)
print(*product_new,sep='\n')
print(*product_ref,sep='\n')

product_new可以正常工作并返回所有产品名称的列表，但是product_ref仅包含重复REF NO. A1400.5类reference出现在网页上的次数。我如何更改此设置以正确存储reference中包含的每次出现在页面上的信息？

谢谢！

Answer 1

在您的代码中，product_name_ref始终是相同的值，因为您是从soup中选择，而不是从caption中选择。

要获取所需信息，可以使用以下示例：

from bs4 import BeautifulSoup


txt = '''
<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup</h2>
   <span class="reference-number">REF NO. A1400.5</span>
</div>

<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup 2</h2>
   <span class="reference-number">REF NO. A1400.5 2</span>
</div>
'''

soup = BeautifulSoup(txt, 'html.parser')

product_new = []
product_ref = []

for product in soup.select('div.product-details'):
    product_new.append(product.h2.get_text(strip=True))
    product_ref.append(product.select_one('span.reference-number').get_text(strip=True))

print(product_new)
print(product_ref)

打印：

['Weekly Roundup', 'Weekly Roundup 2']
['REF NO. A1400.5', 'REF NO. A1400.5 2']

编辑：

product_new = []
product_ref = []

for product in soup.select('div.product-details'):
    n = product.h2
    r = product.select_one('span.reference-number')

    if n and r:
        product_new.append(n.get_text(strip=True))
        product_ref.append(r.get_text(strip=True))

print(product_new)
print(product_ref)

EDIT2：

from bs4 import BeautifulSoup


txt = '''
<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup</h2>
   <span class="reference-number">REF NO. A1400.5</span>
</div>

<div class="product-details">
   <h2 class="product-name" title=" Weekly Roundup"> Weekly Roundup 2</h2>
   <span class="reference-number">REF NO. A1400.6</span>
</div>
'''

soup = BeautifulSoup(txt, 'html.parser')

product_new = []
product_ref = []

for product in soup.select('div.product-details'):
    n = product.h2
    r = product.select_one('span.reference-number')

    if n and r:
        product_new.append(n.get_text(strip=True))
        product_ref.append(r.get_text(strip=True).rsplit(maxsplit=1)[-1])

print(product_new)
print(product_ref)

打印：

['Weekly Roundup', 'Weekly Roundup 2']
['A1400.5', 'A1400.6']

编辑3：

for a, b in zip(product_new, product_ref):
    print('{:<30} {}'.format(a, b))

打印：

Weekly Roundup                 A1400.5
Weekly Roundup 2               A1400.6

Answer 2

尝试更正参考编号的类名，使用下面给出的代码：

[5, 3, 1]
[9, 7, 5, 3, 1]

如何从BeautifulSoup循环中的span标签获取文本

2 个答案: