Question

我正在尝试对博客“ https://blog.feedspot.com/ai_rss_feeds/”进行爬网，并通过其中的所有链接进行爬网，以在每个爬网的链接中查找与人工智能相关的信息。

博客遵循一种模式-它具有多个RSS Feed，并且每个Feed在UI中都有一个称为“站点”的属性。我需要在“站点”属性中获取所有链接。示例：aitrends.com，sciecedaily.com / ...等。在代码中，主div有一个名为“ rss-block”的类，该类还有另一个名为“ data”的嵌套类，每个数据都有多个

标签和

标签包含在其中。 href中的值提供要爬网的链接。我们需要通过抓取上述结构在每个链接中查找与AI相关的文章。

我尝试了以下代码的各种变体，但似乎没什么帮助。

import requests
from bs4 import BeautifulSoup

page = requests.get('https://blog.feedspot.com/ai_rss_feeds/')
soup = BeautifulSoup(page.text, 'html.parser')

class_name='data'

dataSoup = soup.find(class_=class_name)
print(dataSoup)
artist_name_list_items = dataSoup.find('a', href=True)
print(artist_name_list_items)

我甚至在页面中都很难找到链接，更不用说浏览每个链接以在其中刮取与AI相关的文章了。

如果您可以帮助我解决问题的两个部分，那对我来说将是一个很好的学习。有关HTML结构，请参考https://blog.feedspot.com/ai_rss_feeds/的来源。预先感谢！

Answer 1

如您在页面上看到的那样，前20个结果存储在html中。其他的则是从脚本标记中拉出的，您可以对其进行正则表达式以创建67的完整列表。然后循环该列表，并向这些列表发出请求以获取更多信息。我为初始列表填充提供了两个不同的选择器供您选择（第二个-已注释掉-使用:contains-在bs4 4.7.1+中可用）

from bs4 import BeautifulSoup as bs
import requests, re

p = re.compile(r'feed_domain":"(.*?)",')

with requests.Session() as s:
    r = s.get('https://blog.feedspot.com/ai_rss_feeds/')
    soup = bs(r.content, 'lxml')
    results = [i['href'] for i in soup.select('.data [rel="noopener nofollow"]:last-child')]
    ## or use with bs4 4.7.1 + 
    #results = [i['href'] for i in soup.select('strong:contains(Site) + a')]
    results+= [re.sub(r'\n\s+','',i.replace('\\','')) for i in p.findall(r.text)]

    for link in results:
        #do something e.g.
        r = s.get(link)
        soup = bs(r.content, 'lxml')
        # extract info from indiv page

Answer 2

要获取每个块的所有子链接，可以使用soup.find_all：

from bs4 import BeautifulSoup as soup
import requests
d = soup(requests.get('https://blog.feedspot.com/ai_rss_feeds/').text, 'html.parser')
results = [[i['href'] for i in c.find('div', {'class':'data'}).find_all('a')] for c in d.find_all('div', {'class':'rss-block'})]

输出：

[['http://aitrends.com/feed', 'https://www.feedspot.com/?followfeedid=4611684', 'http://aitrends.com/'], ['https://www.sciencedaily.com/rss/computers_math/artificial_intelligence.xml', 'https://www.feedspot.com/?followfeedid=4611682', 'https://www.sciencedaily.com/news/computers_math/artificial_intelligence/'], ['http://machinelearningmastery.com/blog/feed', 'https://www.feedspot.com/?followfeedid=4575009', 'http://machinelearningmastery.com/blog/'], ['http://news.mit.edu/rss/topic/artificial-intelligence2', 'https://www.feedspot.com/?followfeedid=4611685', 'http://news.mit.edu/topic/artificial-intelligence2'], ['https://www.reddit.com/r/artificial/.rss', 'https://www.feedspot.com/?followfeedid=4434110', 'https://www.reddit.com/r/artificial/'], ['https://chatbotsmagazine.com/feed', 'https://www.feedspot.com/?followfeedid=4470814', 'https://chatbotsmagazine.com/'], ['https://chatbotslife.com/feed', 'https://www.feedspot.com/?followfeedid=4504512', 'https://chatbotslife.com/'], ['https://aws.amazon.com/blogs/ai/feed', 'https://www.feedspot.com/?followfeedid=4611538', 'https://aws.amazon.com/blogs/ai/'], ['https://developer.ibm.com/patterns/category/artificial-intelligence/feed', 'https://www.feedspot.com/?followfeedid=4954414', 'https://developer.ibm.com/patterns/category/artificial-intelligence/'], ['https://lexfridman.com/category/ai/feed', 'https://www.feedspot.com/?followfeedid=4968322', 'https://lexfridman.com/ai/'], ['https://medium.com/feed/@Francesco_AI', 'https://www.feedspot.com/?followfeedid=4756982', 'https://medium.com/@Francesco_AI'], ['https://blog.netcoresmartech.com/rss.xml', 'https://www.feedspot.com/?followfeedid=4998378', 'https://blog.netcoresmartech.com/'], ['https://www.aitimejournal.com/feed', 'https://www.feedspot.com/?followfeedid=4979214', 'https://www.aitimejournal.com/'], ['https://blogs.nvidia.com/feed', 'https://www.feedspot.com/?followfeedid=4611576', 'https://blogs.nvidia.com/'], ['http://feeds.feedburner.com/AIInTheNews', 'https://www.feedspot.com/?followfeedid=623918', 'http://aitopics.org/whats-new'], ['https://blogs.technet.microsoft.com/machinelearning/feed', 'https://www.feedspot.com/?followfeedid=4431827', 'https://blogs.technet.microsoft.com/machinelearning/'], ['https://machinelearnings.co/feed', 'https://www.feedspot.com/?followfeedid=4611235', 'https://machinelearnings.co/'], ['https://www.artificial-intelligence.blog/news?format=RSS', 'https://www.feedspot.com/?followfeedid=4611100', 'https://www.artificial-intelligence.blog/news/'], ['https://news.google.com/news?cf=all&hl=en&pz=1&ned=us&q=artificial+intelligence&output=rss', 'https://www.feedspot.com/?followfeedid=4611157', 'https://news.google.com/news/section?q=artificial%20intelligence&tbm=nws&*'], ['https://www.youtube.com/feeds/videos.xml?channel_id=UCEqgmyWChwvt6MFGGlmUQCQ', 'https://www.feedspot.com/?followfeedid=4611505', 'https://www.youtube.com/channel/UCEqgmyWChwvt6MFGGlmUQCQ/videos']]

通过与美丽汤的链接进行网页爬取

2 个答案: