无法使用Rvest包从RSS feed中提取链接

Question

我正试图从RSS feed获得WSJ文章的链接。

提要看起来像这样：

<rss xmlns:wsj="http://dowjones.net/rss/" xmlns:dj="http://dowjones.net/rss/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
<channel>
<title>WSJ.com: World News</title>
<link>http://online.wsj.com/page/2_0006.html</link>
<atom:link type="application/rss+xml" rel="self" href="http://online.wsj.com/page/2_0006.html"/>
<description>World News</description>
<language>en-us</language>
<pubDate>Mon, 09 Sep 2019 10:56:42 -0400</pubDate>
<lastBuildDate>Mon, 09 Sep 2019 10:56:42 -0400</lastBuildDate>
<copyright>Dow Jones & Company, Inc.</copyright>
<generator>http://online.wsj.com/page/2_0006.html</generator>
<docs>http://cyber.law.harvard.edu/rss/rss.html</docs>
<image>
<title>WSJ.com: World News</title>
<link>http://online.wsj.com/page/2_0006.html</link>
<url>http://online.wsj.com/img/wsj_sm_logo.gif</url>
</image>
<item>
<title>
Boris Johnson Promises Oct. 31 Brexit as Law Passes to Rule Out No Deal
</title>
<link>
https://www.wsj.com/articles/boris-johnson-insists-he-wants-a-brexit-deal-despite-no-deal-planning-11568037248
</link>
<description>
<![CDATA[
British Prime Minister Boris Johnson stuck to his pledge that the U.K. would leave the European Union on Oct. 31—even as a bill aimed at preventing the country from leaving on that date without an agreement became law.
]]>
</description>
<content:encoded/>
<pubDate>Mon, 09 Sep 2019 10:46:00 -0400</pubDate>
<guid isPermaLink="false">SB10710731395272083797004585540162284821560</guid>
<category domain="AccessClassName">PAID</category>
<wsj:articletype>U.K. News</wsj:articletype>
</item>
<item>
<title>
Russian Opposition Puts Putin Under Pressure in Moscow Election
</title>
<link>
https://www.wsj.com/articles/russian-opposition-puts-putin-under-pressure-in-moscow-election-11568029495
</link>
<description>
<![CDATA[
Candidates backed by Russia’s opposition won nearly half the seats up for grabs in Moscow’s city elections Sunday, building on a wave of protests that exposed some of the frailties in President Putin’s closely controlled political machine, but failed to make significant inroads in local races elsewhere.
]]>
</description>
<content:encoded/>
<pubDate>Mon, 09 Sep 2019 07:44:00 -0400</pubDate>
<guid isPermaLink="false">SB10710731395272083797004585539862964447000</guid>
<category domain="AccessClassName">PAID</category>
<wsj:articletype>Russia News</wsj:articletype>
</item>

我一直在使用rvest来获取每篇文章的标题，并且可以使用，但是每次链接都返回空白。我已经以几种不同的方式尝试过该代码，但这是最近的尝试：

rm(list=ls())
library(tidyverse)
library(rvest)
setwd("~/wsj/world_news")

wsj_1 <- "wsj-world_news-1568041806.39885.xml" # a file like the example one provided above

test <- wsj_1 %>% read_html() # reading in example file

items <- wsj_1 %>%
  read_html() %>%
  html_nodes('item') # parsing the xml to get each 'item' which is a separate article

title <- items %>% 
  html_nodes('title') %>% 
  html_text()

link <- items %>% 
  html_node('link') %>% 
  html_text()

有人知道为什么我无法显示链接吗？我得到的是<link>而不是URL。

我也无法提取描述标签中的CDATA文本，但这对我来说并不是主要问题。如果我可以找到足够的链接。

Answer 1

没有您正在使用的确切的RSS提要，我将四肢走动，并采取与我可以找到的rss提要类似的样式。如果查看输出html，您将看到链接实际上是下一个兄弟姐妹，因此您可以使用xpath并指定为following-sibling。我使用purrr生成数据帧，并使用str_squish进行输出的一些清理

---

R：

library(rvest)
library(tidyverse)
library(stringr)

wsj_1 <- 'https://feeds.a.dj.com/rss/RSSWorldNews.xml'
nodes <- wsj_1%>%read_html()%>%html_nodes('item')

df <- map_df(nodes, function(item) {

  data.frame(title = str_squish(item%>%html_node('title') %>% html_text()),
             link = str_squish(item%>%html_node(xpath="*/following-sibling::text()") %>%
                    html_text()),
             stringsAsFactors=FALSE)
})

---

Py：

import requests, re
from bs4 import BeautifulSoup as bs
import pandas as pd

r = requests.get('https://feeds.a.dj.com/rss/RSSWorldNews.xml')
soup = bs(r.content, 'lxml')
titles = []; links = [] 

for i in soup.select('item'):
    titles+=[re.sub(r'\n+\s+\t+',' ',i.title.text.strip())]
    links+=[i.link.next_sibling]

df = pd.DataFrame(zip(titles, links), columns = ['Title', 'Link'])
print(df)