Question

我正在尝试通过在线书签服务从我的帐户中抓取数据。带有书签的页面组织如下：

<!DOCTYPE html>
<html lang="en">
<body>
<div id="item1" class="outer_block">
    <div class="title">Bookmark 1</div>
    <div class="link">
        <a href="https://bookmark1.com">https://bookmark1.com</a>
    </div>
    <div class="tags">
        <a href="http://mylink.com/tag1">tag1</a>
        <a href="http://mylink.com/tag2">tag2</a>
    </div>
</div>
<div id="item2" class="outer_block">
    <div class="title">Bookmark 2</div>
    <div class="link">
        <a href="https://bookmark2.com">https://bookmark2.com</a>
    </div>
    <div class="tags">
        <a href="http://mylink.com/tag1">tag1</a>
    </div>
</div>
<div id="item3" class="outer_block">
    <div class="title">Bookmark 3</div>
    <div class="link">
        <a href="https://bookmark3.com">https://bookmark3.com</a>
    </div>
    <div class="tags">
        <a href="http://mylink.com/tag3">tag3</a>
    </div>
</div>
</body>
</html>

对于每个块，我想提取标题，链接和标签。在Python 3.5中，我这样做：

# Import modules
import requests
from lxml import html

# Read the html
# url = 'mylink'
# page = requests.get(url)
# tree = html.fromstring(page.content)
# This is the replicable example
tree = html.fromstring('<!DOCTYPE html><html lang="en"><body><div id="item1" class="outer_block"> <div class="title">Item 1</div> <div class="link"> <a href="https://bookmark1.com">https://bookmark1.com</a> </div> <div class="tags"> <a href="http://mylink.com/tag1">tag1</a> <a href="http://mylink.com/tag2">tag2</a> </div></div><div id="item2" class="outer_block"> <div class="title">Item 2</div> <div class="link"> <a href="https://bookmark2.com">https://bookmark2.com</a> </div> <div class="tags"> <a href="http://mylink.com/tag1">tag1</a> </div></div><div id="item3" class="outer_block"> <div class="title">Item 3</div> <div class="link"> <a href="https://bookmark3.com">https://bookmark3.com</a> </div> <div class="tags"> <a href="http://mylink.com/tag3">tag3</a> </div></div></body></html>')

我使用xpath来提取字符串的模式，比如标题：

titles = tree.xpath('//div[@class="title"]/text()')
print(titles)

[＆＃39; Bookmark 1＆＃39;，＆＃39; Bookmark 2＆＃39;，＆＃39; Bookmark 3＆＃39;]

为了提取标签，我使用相同的原则：

tags = tree.xpath('//div[@class="tags"]//a/text()')
print(tags)

[＆＃39; tag1＆＃39;，＆＃39; tag2＆＃39;，＆＃39; tag1＆＃39;，＆＃39; tag3＆＃39;]

问题是每个链接都有各种标记，因此我无法将数组titles与数组tags相关联。我以为我可以提取每个块，然后分别处理它们：

blocks = tree.xpath('//div[@class="outer_block"]')
block1 = blocks[0]

我不明白的是，当我从block1中提取标签时，它仍会保留原始html的所有标签。

tags_block1 = block1.xpath('//div[@class="tags"]//a/text()'
print(tags_block1)

[＆＃39; tag1＆＃39;，＆＃39; tag2＆＃39;，＆＃39; tag1＆＃39;，＆＃39; tag3＆＃39;]

如何提取标题和相应的标签，最佳输出格式是什么？是否还有其他可以更轻松完成工作的软件包？

Answer 1

你应该考虑使用BeautifulSoup。请考虑以下代码（source是HTML的字符串）：

from bs4 import BeautifulSoup 

soup = BeautifulSoup(source, "html.parser")
outer_blocks = soup.find_all("div", class_="outer_block")

for block in outer_blocks:
    title = block.find("div", class_="title").contents[0]
    link = block.find("a").contents[0]
    tags = [x.contents[0] for x in block.find("div", class_="tags").find_all("a")]
    print([title, link, tags])

输出结果为：

['Bookmark 1', 'https://bookmark1.com', ['tag1', 'tag2']]
['Bookmark 2', 'https://bookmark2.com', ['tag1']]
['Bookmark 3', 'https://bookmark3.com', ['tag3']]

Answer 2

您可以在两个不同的括号中使用两个属性

description = tree.xpath("//div[@class='details-content'][@itemprop='description']/text()")

Python Web抓取：使用多个标记提取一个属性

2 个答案: