Question

我正在尝试从各种页面中提取一些信息，并且有些挣扎。这显示了我的挑战：

import requests
from lxml import html
url = "https://www.soccer24.com/match/C4RB2hO0/#match-summary"
response = requests.get(url)
print(response.content)

如果将输出复制到记事本中，则无法在输出的任何位置找到值“ 9.20”（团队A赔率在网页右下角）。但是，如果打开网页，请执行“另存为”，然后像这样将其导入回Python，则可以找到并提取9.20值：

with open(r'HUL 1-7 TOT _ Hull - Tottenham _ Match Summary.html', "r") as f:
    page = f.read()
tree = html.fromstring(page)

output = tree.xpath('//*[@id="default-odds"]/tbody/tr/td[2]/span/span[2]/span/text()')  #the xpath for the TeamA odds or the 9.20 value
output # ['9.20']

不知道为什么这种解决方法有效，但是那超出了我。因此，我想做的就是将网页保存到本地驱动器中，然后如上所述用Python打开它并从那里继续进行。但是，如何在Python中复制另存为？这不起作用：

import urllib.request
response = urllib.request.urlopen(url)
webContent = response.read().decode('utf-8')
f = open('HUL 1-7 TOT _ Hull - Tottenham _ Match Summary.html', 'w')
f.write(webContent)
f.flush()
f.close()

它给了我一个网页，但这只是原始页面的一小部分...？

Answer 1

正如@Pedro Lobito所说。页面内容由javascript生成。因此，您需要一个可以运行JavaScript的模块。我将选择requests_html或selenium。

Requests_html

from requests_html import HTMLSession

url = "https://www.soccer24.com/match/C4RB2hO0/#match-summary"

session = HTMLSession()
response = session.get(url)
response.html.render()
result = response.html.xpath('//*[@id="default-odds"]/tbody/tr/td[2]/span/span[2]/span/text()')
print(result)
#['9.20']

硒

from selenium import webdriver
from lxml import html

url = "https://www.soccer24.com/match/C4RB2hO0/#match-summary"
dr = webdriver.Chrome()

try:
    dr.get(url)
    tree = html.fromstring(dr.page_source)
    ''' use it when browser closes before loading succeeds
    # https://selenium-python.readthedocs.io/waits.html
    WebDriverWait(dr, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
    '''
    output = tree.xpath('//*[@id="default-odds"]/tbody/tr/td[2]/span/span[2]/span/text()')  #the xpath for the TeamA odds or the 9.20 value
    print(output)

except Exception as e:
    raise e

finally:
    dr.close()
#['9.20']

保存麻烦的网页并重新导入Python

1 个答案: