无法从eBay的Beautiful Soup中获取数据

Question

我正在尝试从ebay抓取数据，当我试图找到一个div时，它总是给我一个None。我的代码有什么错误。

from bs4 import BeautifulSoup
from selenium import webdriver
from openpyxl import Workbook
wb = Workbook()
ws = wb.active

driver = webdriver.Chrome(executable_path = r'E:\chromedriver\chromedriver.exe')

urlBase = "https://www.ebay.com.au/str/Tyroola?rt=nc&_pgn="

for i in range(2, 3):
    url = urlBase + str(i)
    driver.get(url)
    pagehtml = driver.page_source
    soup = BeautifulSoup(pagehtml, 'lxml')

    tyreList = soup.find_all('li', class_='s-item')
    for listItem in tyreList:
        tyre = listItem.find('a')
        driver.get(tyre['href'])
        ebayPage = driver.page_source
        ebaySoup = BeautifulSoup(ebayPage, 'lxml')
        tableDiv = ebaySoup.find('div', id='viTabs_0_is')
        proDiv = ebaySoup.find('div', class_ = 'product-details')
        print(proDiv)

proDiv始终为无。

Answer 1

eBay website上出现的项目详细信息以HTML表格的形式出现。 总共存在5行，其中恰好存在2个<td>标签。 <td>标签定义表中的单元格 这是您可以在Beautiful Soup中解析此表的方法之一。

from bs4 import BeautifulSoup
from selenium import webdriver
from openpyxl import Workbook
wb = Workbook()
ws = wb.active

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")

urlBase = "https://www.ebay.com.au/str/Tyroola?rt=nc&_pgn="

for i in range(2, 3):
    url = urlBase + str(i)
    driver.get(url)
    pagehtml = driver.page_source
    soup = BeautifulSoup(pagehtml, 'lxml')

    tyreList = soup.find_all('li', class_='s-item')
    for listItem in tyreList:
        tyre = listItem.find('a')
        driver.get(tyre['href'])
        ebayPage = driver.page_source
        ebaySoup = BeautifulSoup(ebayPage, 'lxml')
        item_specifics_table = ebaySoup.find('div', class_='section') #table starts here
        rows = item_specifics_table.findAll('tr')#finding all the content of tr tag and saving it into a list:rows
        print
        for cells in rows:#to parse td tags we need to look into each index of the list:rows
            print "{0} {1}\n{2} {3}".format(cells.contents[1].string.strip(), cells.contents[3].find('span').get_text().strip(),
                                            cells.contents[5].string.strip(), cells.contents[7].find('span').get_text().strip())
        # strip() removes excessive whitespace
        print"********************************************************************************"
        print

Answer 2

阅读有关 scraping eBay using BeautifulSoup in Python 的完整指南，以消除混乱。 它涵盖以下主题：

1。选择所需的信息

1. 最终确定要提取的标签
2. 以结构化格式放置抓取的数据
3. 可视化结果（可选）
4. 所需的库和安装
5. 用于抓取eBay（完整代码集）的Python实现
6. 可视化产品价格