Question

我正在尝试从此网站检索图像（经许可）。以下是我想要访问的网站的代码：

import urllib2
from bs4 import BeautifulSoup

url = "http://www.vgmuseum.com/nes.htm"

page = urllib2.urlopen(url).read()
soup = BeautifulSoup(page, "html5lib")
li = soup.select('ol > li > a')
for link in li:
   print(link.get('href'))

我想要使用的图像在此有序列表中： list location for images

Answer 1

您正在使用的页面由iframe组成，这基本上是将一个页面包含在另一个页面中的一种方式。浏览器了解iframe的工作方式，并将下载页面并在浏览器窗口中显示它们。

但是，

urllib2不是浏览器，不能这样做。您需要浏览链接列表所在的位置，iframe，然后按照此iframe内容来源的网址进行操作。在您的情况下，左侧的链接列表来自http://www.vgmuseum.com/nes_b.html页面。

这是一个有效的解决方案，可以跟踪链接列表中的链接，将包含图像的页面和downloading images下载到images/目录中。我正在使用requests模块并利用lxml解析器与BeautifulSoup合作以加快HTML解析速度：

from urllib.parse import urljoin

import os
import requests
from bs4 import BeautifulSoup

url = "http://www.vgmuseum.com/nes_b.html"


def download_image(session, url):
    print(url)
    local_filename = os.path.join("images", url.split('/')[-1])

    r = session.get(url, stream=True)
    with open(local_filename, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk:  # filter out keep-alive new chunks
                f.write(chunk)


with requests.Session() as session:
    session.headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36'
    }
    response = session.get(url)
    soup = BeautifulSoup(response.content, "lxml")

    for link in soup.select('ol > li > a[href*=images]'):
        response = session.get(urljoin(response.url, link.get('href')))
        for image in BeautifulSoup(response.content, "lxml").select("img[src]"):
            download_image(session, url=urljoin(response.url, image["src"]))

Answer 2

我使用上面@ Dan的评论中的网址进行解析。

代码：

import requests
from bs4 import BeautifulSoup

url = 'http://www.vgmuseum.com/nes_b.html'

page = requests.get(url).text
soup = BeautifulSoup(page, 'html.parser')
li = soup.find('ol')
soup = BeautifulSoup(str(li), 'html.parser')
a = soup.find_all('a')
for link in a:
    if not link.get('href') == '#top' and not link.get('href') == None:
        print(link.get('href'))

输出：

images/nes/10yard.html
images/nes2/10.html
pics2/100man.html
images/nes/1942.html
images/nes2/1942.html
images/nes/1943.html
images/nes2/1943.html
pics7/1944.html
images/nes/1999.html
images/nes2/2600.html
images/nes2/3dbattles.html
images/nes2/3dblock.html
images/nes2/3in1.html
images/nes/4cardgames.html
pics2/4.html
images/nes/4wheeldrivebattle.html
images/nes/634.html
images/nes/720NES.html
images/nes/8eyes.html
images/nes2/8eyes.html
images/nes2/8eyesp.html
pics2/89.html
images/nes/01/blob.html
pics5/boy.html
images/03/a.html
images/03/aa.html
images/nes/abadox.html
images/03/abadoxf.html
images/03/abadoxj.html
images/03/abadoxp.html
images/03/abarenbou.html
images/03/aces.html
images/03/action52.html
images/03/actionin.html
images/03/adddragons.html
images/03/addheroes.html
images/03/addhillsfar.html
images/03/addpool.html
pics/addamsfamily.html
pics/addamsfamilypugsley.html
images/nes/01/adventureislandNES.html
images/nes/adventureisland2.html
images/nes/advisland3.html
pics/adventureisland4.html
images/03/ai4.html
images/nes/magickingdom.html
pics/bayou.html
images/03/bayou.html
images/03/captain.html
images/nes/adventuresofdinoriki.html
images/03/ice.html
images/nes/01/lolo1.html
images/03/lolo.html
images/nes/01/adventuresoflolo2.html
images/03/lolo2.html
images/nes/adventuresoflolo3.html
pics/radgravity.html
images/03/rad.html
images/nes/01/rockyandbullwinkle.html
images/nes/01/tomsawyer.html
images/03/afroman.html
images/03/afromario.html
pics/afterburner.html
pics2/afterburner2.html
images/03/ai.html
images/03/aigiina.html
images/nes/01/airfortress.html
images/03/air.html
images/03/airk.html
images/nes/01/airwolf.html
images/03/airwolfe.html
images/03/airwolfj.html
images/03/akagawa.html
images/nes/01/akira.html
images/03/akka.html
images/03/akuma.html
pics2/adensetsu.html
pics2/adracula.html
images/nes/01/akumajo.html
pics2/aspecial.html
pics/alunser.html
images/nes/01/alfred.html
images/03/alice.html
images/nes/01/alien3.html
images/nes/01/asyndrome.html
images/03/alien.html
images/03/all.html
images/nes/01/allpro.html
images/nes/01/allstarsoftball.html
images/nes/01/alphamission.html
pics2/altered.html

使用BeautifulSoup从有序列表中刮取图像hrefs

2 个答案: