Question

该脚本的目的是访问一个网站，该网站然后通过get_attribute使用硒生成所有产品的链接列表。

我使用请求访问这些新生成的链接，以访问每种产品。然后，我尝试使用在不同特征变量中存储的BeautifulSoup进行抓取。

我的问题是，我认为我要刮擦的某些产品没有我要刮擦的类别，但是，我相信其中大多数都可以。对于没有我要存储的存储特征的产品，是否可以返回“ N / A”之类的信息？

这是我的代码：

import time
import csv
from selenium import webdriver
import selenium.webdriver.chrome.service as service
import requests
from bs4 import BeautifulSoup

all_product = []

url = "https://www.vatainc.com/infusion.html?limit=all"
service = service.Service('/Users/Jonathan/Downloads/chromedriver.exe')
service.start()
capabilities = {'chrome.binary': '/Google/Chrome/Application/chrome.exe'}
driver = webdriver.Remote(service.service_url, capabilities)
driver.get(url)
time.sleep(2)
links = [x.get_attribute('href') for x in driver.find_elements_by_xpath("//*[contains(@class, 'product-name')]/a")]

for link in links:
    html = requests.get(link).text
    soup = BeautifulSoup(html, "html.parser")
    products = soup.findAll("html")

    for product in products:
        name = product.find("div", {"class": "product-name"}).text.strip('\n\r\t": ')
        manufacturing_SKU = product.find("span", {"class": "i-sku"}).text.strip('\n\r\t": ')
        manufacturer = product.find("p", {"class": "manufacturer"}).text.strip('\n\r\t": ')
        description = product.find("div", {"class": "std description"}).text.strip('\n\r\t": ')
        included_products = product.find("div", {"class": "included_parts"}).text.strip('\n\r\t": ')
        price = product.find("span", {"class": "price"}).text.strip('\n\r\t": ')
        all_product.append([name, manufacturing_SKU, manufacturer, description, included_products, price])
print(all_product)

这是我的错误代码：

 AttributeError                            Traceback (most recent call last)
<ipython-input-25-36feec64789d> in <module>()
     34         manufacturer = product.find("p", {"class": "manufacturer"}).text.strip('\n\r\t": ')
     35         description = product.find("div", {"class": "std description"}).text.strip('\n\r\t": ')
---> 36         included_products = product.find("div", {"class": "included_parts"}).text.strip('\n\r\t": ')
     37         price = product.find("span", {"class": "price"}).text.strip('\n\r\t": ')
     38         all_product.append([name, manufacturing_SKU, manufacturer, description, included_products, label, price])

AttributeError: 'NoneType' object has no attribute 'text'

Answer 1

find()对象上的BeautifulSoup方法在找不到与查询匹配的DOM元素时返回None。具体来说，在该行included_products上，它找不到类为div的{{1}}。

在这种情况下，您可以执行以下操作以获取included_parts的{{1}}值：

included_products

如何使用BeautifulSoup刮

1 个答案: