XKCD漫画抓取程序出错

时间：2017-03-02 14:19:06

标签： python python-3.x

按照“自动无聊的东西”一书，我写了一个脚本来下载每个xkcd漫画。我完全按照书中的代码（据我所知），但我收到了一个我不明白的错误。 Link 回溯：

/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 /Users/ericdusseau/PycharmProjects/AutomateTheBoringStuff/downloadXkcd.py
Downloading page http://xkcd,com...
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 743, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 356, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 166, in connect
    conn = self._new_conn()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.HTTPConnection object at 0x104f80240>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 649, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/util/retry.py", line 376, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='xkcd,com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x104f80240>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ericdusseau/PycharmProjects/AutomateTheBoringStuff/downloadXkcd.py", line 11, in <module>
    res = requests.get(url)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='xkcd,com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x104f80240>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

Process finished with exit code 1

源码：

#! python3
# downloadXKCD.py - Downloads every single XKCD comic.

import requests, os, bs4

url = 'http://xkcd,com'             #starting url
os.makedirs('xkcd', exist_ok=True)  # store comics in ./xkcd
while not url.endswith('#'):
    #Download the page
    print('Downloading page %s...' % url)
    res = requests.get(url)
    res.raise_for_status()

    soup = bs4.BeautifulSoup(res.text)

    #Find the URL of the comic image.
    comicElem = soup.select('#comic img')
    if comicElem == []:
        print('Could not find comic image.')
    else:
        try:
            comicUrl = 'http:' + comicElem[0].get('src')
            #Download the image.
            print('Downloading image %s...' % (comicUrl))
            res = requests.get(comicUrl)
            res.raise_for_status()
        except requests.exceptions.MissingSchema:
            #Skip this comic
            prevLink = soup.select('a[rel="prev"]')[0]
            url = 'http://xkcd.com' + prevLink.get('href')
            continue


    #Save the image to ./xkcd
    imageFile = open(os.path.join('xkcd', os.path.basename(comicUrl)), 'wb')
    for chunk in res.iter_content(100000):
        imageFile.write(chunk)
    imageFile.close()


    #Get the Prev button's url.
    prevLink = soup.select('a[rel="prev"]')[0]
    url = 'htt[://xkcd.com' + prevLink.get('href')

print('Done.')

4 个答案:

答案 0 :(得分：4)

您的输出可以让您了解问题所在：

Downloading page http://xkcd,com...
Traceback (most recent call last):

http://xkcd,com是无效的网址。它应该是.com，而不是,com。

说到错误的网址，您的代码后面还会出现另一个无效网址：

url = 'htt[://xkcd.com' + prevLink.get('href')
          ^ Should be 'p'

答案 1 :(得分：1)

I made a few changes to the code in "Automate the boring .." book and dont see issues even if there are no images try: elem = requests.get('https://xkcd.com/') elem.raise_for_status() except Exception as esc: print('exception in search:%s' %(esc))

elemSoup = bs4.BeautifulSoup(elem.text,'html.parser')

elemSouplink = elemSoup.select('.comicNav a') # print('elements:'+str(elemSouplink))

while (not (elemSouplink[1].get('href')=='/1/')):

   prev= elemSouplink[1].get('href')
  # print('prev:'+str(prev))
   elem=requests.get('https://xkcd.com'+prev)
  # print('new link:https://xkcd.com'+prev)
   elemSoup = bs4.BeautifulSoup(elem.text,'html.parser')
   elemSouplink = elemSoup.select('.comicNav a')

答案 2 :(得分：0)

xkcd,com中有一个逗号。不是一个点......

答案 3 :(得分：0)

有一个git存储库，其中包含一个下载随机xkcd漫画图像并将该图像设置为墙纸的脚本。你可以在每分钟在cron中设置它，以便它将改变桌面壁纸：

xkcd_random_wallpaper

相关问题
最新问题