如何从HTML标题中获取引用的字符串？

Question

鉴于此HTML片段，如何在href =？之后使用python包request或xlml查找引用的字符串？

<dl>
    <dt><a href="oq-phys.htm">
        <b>Physics and Astronomy</b></a>
    <dt><a href="oq-math.htm">
        <b>Mathematics</b></a>
    <dt><a href="oq-life.htm">
        <b>Life Sciences</b></a>
    <dt><a href="oq-tech.htm">
        <b>Technology</b></a>
    <dt><a href="oq-geo.htm">
        <b>Earth and Environmental Science</b></a>
</dl>

Answer 1

  

在href=

之后找到引用的字符串

短requests + beautifulsoup解决方案：

import requests, bs4

soup = bs4.BeautifulSoup(requests.get('http://.openquestions.com').content, 'html.parser')
hrefs = [a['href'] for a in soup.select('dl dt a')]
print(hrefs)

输出：

['oq-phys.htm', 'oq-math.htm', 'oq-life.htm', 'oq-tech.htm', 'oq-geo.htm', 'oq-map.htm', 'oq-about.htm', 'oq-howto.htm', 'oqc/oqc-home.htm', 'oq-indx.htm', 'oq-news.htm', 'oq-best.htm', 'oq-gloss.htm', 'oq-quote.htm', 'oq-new.htm']

Answer 2

对于上面的示例，假设我们有包含上述代码段的html_string。

import requests
import lxml.etree as LH
html_string =  LH.fromstring(requests.get('http://openquestions.com').text)

for quoted_link in html_string.xpath('//a'): print(quoted_link.attrib['href'], quoted_link.text_content())

Answer 3

这种猫的皮肤会有很多方法。这是一个requests / lxml解决方案，不包含（显式）for循环：

import requests
from lxml.html import fromstring

req = requests.get('http://www.openquestions.com')
resp = fromstring(req.content)
hrefs = resp.xpath('//dt/a/@href') 
print(hrefs)

修改

为什么我这样写：

• 我更喜欢XPath到CSS选择器
• 很快

基准：

import requests,bs4
from lxml.html import fromstring
import timeit

req = requests.get('http://www.openquestions.com').content

def myfunc() :
    resp = fromstring(req)
    hrefs = resp.xpath('//dl/dt/a/@href')

print("Time for lxml: ", timeit.timeit(myfunc, number=100))

##############################################################

resp2 = requests.get('http://www.openquestions.com').content

def func2() :
    soup = bs4.BeautifulSoup(resp2, 'html.parser')
    hrefs = [a['href'] for a in soup.select('dl dt a')]

print("Time for beautiful soup:", timeit.timeit(func2, number=100))

输出：

('Time for lxml: ', 0.09621267095780464)
('Time for beautiful soup:', 0.8594218329542824)