Question

我试图使用lxml解析下面的网页。但是我的xpath似乎有些不对劲。我不确定我做错了什么。

web_content = requests.get(r"https://www.quandl.com/data/TSE").content
dataset_count = html.fromstring(web_content)
print(dataset_count.xpath(r'//*[@id="ember667"]/div[2]/main/section/section/section[2]/div[3]/div[2]/span[2]'))

我试图让它返回3908的数据集编号。但是这个xpath似乎对我不起作用。有什么想法吗？

另外，我希望如果我通过请求传递另一个quandl链接，我可以使用相同的xpath来绘制数据集编号。那可能吗？

Answer 1

数据集计数似乎也在<noscript>元素中：

<div class='centered' id='main' role='main'>
<div id='content'>
<noscript>
<table>
<tbody>
<tr>
<td>Database Name</td>
<td>Tokyo Stock Exchange</td>
</tr>
<tr>
<td></td>
<td></td>
</tr>
<tr>
<td>Datasets</td>
<td>3908</td>
</tr>
<tr>
<td>Downloads</td>
<td>4067259</td>
</tr>
<tr>
...

所以你可以用这样的东西抓住它：

>>> import requests
>>> import lxml.html

>>> r = requests.get('https://www.quandl.com/data/TSE')
>>> h = lxml.html.fromstring(r.text)
>>> h
<Element html at 0x7ffb5f6ed0a8>

>>> h.xpath('//noscript')
[<Element noscript at 0x7ffb5c16ac58>, <Element noscript at 0x7ffb5c16ac00>]

>>> h.xpath('string(//noscript//tr[td[1]="Datasets"]/td[2])')
'3908'
>>> h.xpath('string(//div[@id="content"]//noscript//tr[td[1]="Datasets"]/td[2])')
'3908'
>>> h.xpath('number(//div[@id="content"]//noscript//tr[td[1]="Datasets"]/td[2])')
3908.0

OP请求的XPath说明：

//div[@id="content"]          <-- look for a <div> element with "id" attribute equal to "content"
  //noscript                  <-- look for a <noscript> descendant
    //tr[                     <-- look for a <tr> descendant...
        td[1]="Datasets"      <-- ... which 1st <td> child string value is "Datasets"...
                              (this is true if the <td> contains only 1 text node "Datasets"
        ]
      /td[2]                  <-- select the 2nd <td> of previous matching <tr> rows

Answer 2

requests收到的响应中没有3908号码，因为号码是通过附加请求动态加载的。

解决此问题的一个选择是使用真实的浏览器并使用selenium控制它。以下是使用PhantomJS headless browser的示例工作代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


driver = webdriver.PhantomJS()
driver.get("https://www.quandl.com/data/TSE")

wait = WebDriverWait(driver, 10)
elm = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, ".database-statistics .column:nth-child(2) span:nth-child(2)")))
print(elm.text)

driver.close()

打印3,908。

lxml xpath无法显示html项目

2 个答案: