Question

我是Python的新手，并且已经使用bs4学习了Web Scraping的基础。在这里，我尝试提取Youtube搜索结果的所有链接，但在其他网站上却无法正常工作。我分析了搜索结果html数据，搜索结果的链接位于ID为“视频标题”的锚标记中，但该标记未出现在我的bs4解析的html文档中

from bs4 import BeautifulSoup as bs
import requests
name=input("Enter video name ")
url='https://www.youtube.com/results?search_query='+name
searched=requests.get(url)
soup=bs(searched.text,'html.parser')
aid=soup.find_all('a',{'id':'video-title'})
print(aid)

我希望输出包含所有搜索结果。我还没有学过其他软件包，如果可能的话，我想在bs4中做到这一点。

Answer 1

获取YouTube搜索结果数据的所有麻烦只是浪费时间和精力。

为什么不尝试这些选项

也就是说，对于前20个结果，您可以从源代码中的JavaScript内容中获取数据。答案如下。

大约1个小时理解了生成的json后，对于某些查询仍然失败。YouTube是一个非常复杂的网站。响应可能会因位置，浏览器，搜索查询等而异。

我们正在从源代码中的这个脚本标签中提取数据。

代码：

from bs4 import BeautifulSoup as bs
import requests
import re
import json
headers={
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36'
}
name=input("Enter video name: ")
url='https://www.youtube.com/results?search_query=hello'+name
searched=requests.get(url,headers=headers)
soup=bs(searched.text,'html.parser')
aid=soup.find('script',string=re.compile('ytInitialData'))
extracted_josn_text=aid.text.split(';')[0].replace('window["ytInitialData"] =','').strip()
video_results=json.loads(extracted_josn_text)
#print(item_section=video_results["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"][1])
item_section=video_results["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"][0]["itemSectionRenderer"]["contents"]

for item in item_section:
    try:
        video_info=item["videoRenderer"]
        title=video_info["title"]["simpleText"]
        url=video_info["navigationEndpoint"]["commandMetadata"]["webCommandMetadata"]["url"]
        print('Title:',title)
        print('Url:',url, end="\n----------\n")
    except KeyError:
            pass

输出：

Enter video name: hello
Title: New Punjabi Songs 2017-Hello Hello(Ful Song)-Prince Narula-Yuvika Chaudhary-Latest Punjabi Song 2017
Url: /watch?v=mv326-zVpAQ
----------
Title: Alan Walker - The Spectre
Url: /watch?v=wJnBTPUQS5A
----------
Title: Hello Hello (Full HD) - Rajvir Jawanda | MixSingh | Josan Bros | New Punjabi Songs 2018
Url: /watch?v=xydupjQSj44
----------
Title: Bachchan - Hello Hello - Kannada Movie Full Song Video | Sudeep | Bhavana | V Harikrishna
Url: /watch?v=oLMMgoug4Uk
----------
Title: Hello Hello latest 2017 16 june punjabi song
Url: /watch?v=MqCSsPXw8QU
----------
Title:   Hello Hello   | + More Kids Songs | Super Simple Songs
Url: /watch?v=saDkICxEdgY
----------
Title: 'Gallan Goodiyaan' Full VIDEO Song | Dil Dhadakne Do | T-Series
Url: /watch?v=jCEdTq3j-0U
----------
Title: Hello Hello Gippy Grewal Feat. Dr. Zeus Full Song HD | Latest Punjabi Song 2013
Url: /watch?v=IRW2O4QZhgs
----------
Title: Hello Hello | Pataakha | Malaika Arora | Vishal Bhardwaj & Rekha Bhardwaj | Gulzar | Ganesh Acharya
Url: /watch?v=RxBAitQLSLA
----------
Title: Hello Hello (Lyrical Audio) Prince Narula ft. Yuvika Chaudhary | Punjabi Lyrical Audio 2017 | WHM
Url: /watch?v=v8VIsIvhDoQ
----------
Title: Hello Hello Full Video Song || Bhale Bhale Magadivoi || Nani, Lavanya Tripathi
Url: /watch?v=y3FI02OO_kU
----------
Title: Hello hello gaad bahe dhufee na egaa  (new comedy hhhhhh)
Url: /watch?v=DuRrcTo4rgg
----------
Title: Proper Patola - Official Video | Namaste England | Arjun | Parineeti | Badshah | Diljit | Aastha
Url: /watch?v=YmXJp4RtBCM
----------
Title: Official Video: Nikle Currant Song | Jassi Gill | Neha Kakkar | Sukh-E Muzical Doctorz | Jaani
Url: /watch?v=uBaqgt5V0mU
----------
Title: Insane (Full Song)  Sukhe - Jaani - Arvindr Khaira - White Hill Music - Latest Punjabi Song 2018
Url: /watch?v=mKpPhVVF8So
----------
Title: Radha bole HELLO HELLO-cartoon song mix with step up 2
Url: /watch?v=TFCTgNCzrck
----------
Title: Hello Song | CoCoMelon Nursery Rhymes & Kids Songs
Url: /watch?v=fxVMqaViVaA
----------
Title: Bachchan - Hello Hello Unplugged Version | Sudeep | Bhavana | V Harikrishna
Url: /watch?v=lvH3kTGJeEQ
----------
Title: Hello Hello! Can You Clap Your Hands? | Original  Kids Song | Super Simple Songs
Url: /watch?v=fN1Cyr0ZK9M
----------

您可以尝试的最后一件事是模拟youtube本身使用的API

即。向

发送POST请求

https://www.youtube.com/results?search_query=yoursearchtext

它有很多cookie和会话值作为参数发送。您可能需要模拟所有这些。您可能需要使用Requests session objects来做到这一点。

提取YouTube搜索结果时出现问题

1 个答案: