BeautifulSoup

Question

我正在将网页加载到iframe中，我想确保将所有相关媒体都可用。我目前正在使用请求下载页面，然后进行一些查找/替换，但这并没有全面覆盖。有没有办法用python来获取页面加载到浏览器时所做的所有脚本，CSS和图像请求的列表？

Answer 1

BeautifulSoup

使用BeautifulSoup4获取所有<img>，<link>和<script>标记，然后提取相应的属性。

from bs4 import BeautifulSoup
import requests

resp = requests.get("http://www.yahoo.com")

soup = BeautifulSoup(resp.text)

# Pull the linked images (note: will grab base64 encoded images) 
images = [img['src'] for img in soup.findAll('img') if img.has_key('src')]

# Checking for src ensures that we don't grab the embedded scripts
scripts = [script['src'] for script in soup.findAll('script') if script.has_key('src')]

# favicon.ico and css
links = [link['href'] for link in soup.findAll('link') if link.has_key('href')]

示例输出：

In [30]: images = [img['src'] for img in soup.findAll('img') if img.has_key('src')]

In [31]: images[:5]
Out[31]:
['http://l.yimg.com/dh/ap/default/130925/My_Yahoo_Defatul_HP_ad_300x250.jpeg',
 'http://l.yimg.com/os/mit/media/m/base/images/transparent-95031.png',
 'http://l.yimg.com/os/mit/media/m/base/images/transparent-95031.png',
 'http://l.yimg.com/os/mit/media/m/base/images/transparent-95031.png',
 'http://l.yimg.com/os/mit/media/m/base/images/transparent-95031.png']

使用Python中的请求或机械加载所有第三方脚本

1 个答案:

BeautifulSoup