Question

我正在尝试从NFL Teams网站上抓取一些数据，并且在python输出中不断收到空字符串[]。

我正在尝试获取团队名称以及链接。

这是我要在findAll()上使用的HTML：

<td customsortid="fullName" id="standingsTile" data="[object Object]" class="rmq-30d300f6" data-radium="true" style="border-top: 1px solid rgb(238, 238, 238); box-sizing: border-box; height: auto; line-height: 40px; padding-left: 16px; padding-right: 16px; text-align: right; white-space: nowrap; vertical-align: top; box-shadow: rgb(238, 238, 238) 4px 0px 1px; left: 0px; position: absolute; width: 160px;">
    <a href="http://www.nfl.com/teams/profile?team=NE" tabindex="0" data-metrics-link-module="divisionStandings0" data-radium="true" style="cursor: pointer; text-decoration: none;">
    <div class="rsv-ae9db127" data-radium="true">
        <div class="rsv-ae9db127" data-radium="true" style="align-items: center; padding: 0px; background-color: transparent; height: 40px; -webkit-box-align: center;">
            <div data-radium="true">
                <img alt="" data-test-id="facemask-image-container" sizes="100vw" src="https://static.nfl.com/static/content/public/static/wildcat/assets/img/logos/teams/NE.svg" width="48" data-radium="true" style="border: 0px; display: block; max-width: 100%; margin-right: 8px; width: 24px;">
            </div>
            <div class="rsv-ae9db127" data-radium="true">
                <div class="rmq-9da922a7" data-radium="true" style="line-height: 1; font-size: 12px; margin-bottom: 0px; color: rgb(51, 51, 51); text-transform: none; font-family: &quot;Endzone Sans Medium&quot;, sans-serif; display: none; margin-right: 4px; text-decoration: none;">
                    New England 
                </div>
                <div data-radium="true" style="color: rgb(51, 51, 51); font-family: &quot;Endzone Sans Medium&quot;, sans-serif; font-size: 12px; line-height: 1; margin-top: 0px; text-decoration: none; text-transform: none;">
                    Patriots
                </div>
            </div>
        </div>
        <div class="rsv-ae9db127" data-radium="true" style="font-size: 10px; line-height: 1; padding-left: 4px; padding-top: 8px;">
            z
        </div>
    </div>
    </a>
</td>

这是我的代码，总是给我一个空白列表[]

from bs4 import BeautifulSoup as bsoup
from urllib.request import urlopen as uReq

nfl_url = ("https://www.nfl.com/standings")
webpage = uReq(nfl_url)
page_html = webpage.read()
page_parser = bsoup(page_html , "html.parser")
odd = page_parser.findAll("td", {"class": "rmq-30d300f6"})
print(odd)

我要从哪里获取团队名称和链接 https://www.nfl.com/standings

关于我在做什么错的任何想法？

Answer 1

您的代码可以与示例HTML一起正常工作。但是，当重新获取URL时，它没有相同的数据。也许页面是用javascript呈现的，并且服务器未提供您在示例中放置的实际HTML。提取页面后，保存一份副本供自己比较：

...
page_html = webpage.read()

print("Read %d bytes" % (len(page_html)))
open("some.html","wt").write(page_html.decode('utf-8'))
...

保存到“ some.html”的内容与您期望的内容明显不同。

也许您可以从服务器返回的JavaScript输入数据中解析所需的信息：

...
{"conference":"AMERICAN_FOOTBALL_CONFERENCE","division":"AFC_EAST","teamId":"10043200-2018-239d-5857-a43b18004fb2","fullName":"New England Patriots","nickName":"Patriots","overallWin":11,"overallLoss":5  //etc.
...

Answer 2

您要获取的HTML内容是由Javascript生成的，因此不在您从网站抓取的原始HTML中。

您应该考虑使用Selenium加载页面，并等待直到加载所需的元素。像这样：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

from bs4 import BeautifulSoup as bsoup


nfl_url = "https://www.nfl.com/standings"
browser = webdriver.Chrome('/home/your_username/Downloads/chromedriver')
browser.get(nfl_url)
delay = 10
try:
    # Using 'fullName' ID as a check that the relevant portion of the page is loaded.
    # There may be other elements that are more appropriate - up to you to figure out.
    myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'fullName')))
except TimeoutException:
    print( "Loading took too much time!")

page_html = browser.page_source
browser.close()

page_parser = bsoup(page_html , "html.parser")
odd = page_parser.findAll("td", {"class": "rmq-30d300f6"})
print(odd)

请注意，您将需要指定chromedriver的路径，如果尚未在您的系统上，请指定download it。

Python抓取问题

2 个答案: