Question

我正在尝试使用Python进行图像抓取。一次图像抓取成功，但多次图片抓取失败。

#-*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import urllib.request
import random
from array import*

def download_image(url):
    name = random.randrange(1, 1000)
    full_name = str(name) + ".jpg"
    urllib.request.urlretrieve(url, full_name)

if __name__ == "__main__":
    print("Crawling!!!!!!!!!!!!!!!")

    hdr = {'User-Agent': 'Mozilla/5.0', 'referer' : 'http://m.naver.com'}
    req = urllib.request.Request("https://www.google.co.kr/search hl=ko&site=imghp&tbm=isch&source=hp&biw=1600&bih=770&q=sad",headers=hdr)
    data = urllib.request.urlopen(req).read()

    bs = BeautifulSoup(data, 'html.parser')
    imgs = bs.findAll(name='img')

    for img in imgs:
        temp = img.get('src')
        download_image(temp)

这是错误：

Crawling!!!!!!!!!!!!!!!
Traceback (most recent call last):
  File "C:/Users/ajh46/PycharmProjects/untitled1/Crawling.py", line 25, in 
<module>
    download_image(temp)
  File "C:/Users/ajh46/PycharmProjects/untitled1/Crawling.py", line 10, in 
download_image
    urllib.request.urlretrieve(url, full_name)
  File "C:\Users\ajh46\AppData\Local\Programs\Python\Python36-
32\lib\urllib\request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "C:\Users\ajh46\AppData\Local\Programs\Python\Python36-
32\lib\urllib\request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\ajh46\AppData\Local\Programs\Python\Python36-
32\lib\urllib\request.py", line 511, in open
    req = Request(fullurl, data)
  File "C:\Users\ajh46\AppData\Local\Programs\Python\Python36-
 32\lib\urllib\request.py", line 329, in __init__
    self.full_url = url
  File "C:\Users\ajh46\AppData\Local\Programs\Python\Python36-
32\lib\urllib\request.py", line 355, in full_url
    self._parse()
  File "C:\Users\ajh46\AppData\Local\Programs\Python\Python36-
32\lib\urllib\request.py", line 384, in _parse
    raise ValueError("unknown url type: %r" % self.full_url)

Answer 1

目标URL返回一个404。

urllib.request出于这个原因引发错误。

Image如何使用python 3.xx进行爬网

1 个答案: