Question

我正在尝试使用urllib.request.urlretrieve和多处理模块来下载一些文件并对它们进行一些处理。但是，每次我尝试运行我的程序时，都会给出错误：

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "./thumb.py", line 13, in download_and_convert
    filename, headers = urlretrieve(url)
  File "/usr/lib/python3.4/urllib/request.py", line 186, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.4/urllib/request.py", line 463, in open
    response = self._open(req, data)
  File "/usr/lib/python3.4/urllib/request.py", line 486, in _open
    'unknown_open', req)
  File "/usr/lib/python3.4/urllib/request.py", line 441, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.4/urllib/request.py", line 1252, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: http>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./thumb.py", line 27, in <module>
    pool.map(download_and_convert, enumerate(csvr))
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 260, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
urllib.error.URLError: <urlopen error unknown url type: http>

它似乎窒息的网址是http://phytoimages.siu.edu/users/vitt/10_27_06_2/Equisetumarvense.JPG。这是我的代码：

#!/usr/bin/env python3

from subprocess import Popen
from sys import argv, stdin
import csv
from multiprocessing import Pool
from urllib.request import urlretrieve

def download_and_convert(args):
    num, url_list = args
    url = url_list[0]
    try:
        filename, headers = urlretrieve(url)
    except:
        print(url)
        raise
    Popen(["convert", filename, "-resize", "250x250",\
           str(num)+'.'+url.split('.')[-1]])

if __name__ == "__main__":
    csvr = csv.reader(open(argv[1]))

    if(len(argv) > 2): nprocs = argv[2]
    else: nprocs = None

    pool = Pool(processes=nprocs)
    pool.map(download_and_convert, enumerate(csvr))

我不知道为什么会出现这种错误。可能是因为我正在使用多处理？如果有人能帮助我，我将不胜感激。

编辑：这是它尝试处理的第一个网址，如果我更改它，它不会更改错误。

Answer 1

检查此代码段

>>> import urllib.parse
>>> urllib.parse.quote(':')
'%3A'

正如您所见，urllib解释了＆＃39;：＆＃39;性格奇怪。巧合的是你的程序挂在你身上。

尝试urllib.parse.urlencode()，让您走上正确的轨道。

Answer 2

在评论的一些帮助下，我找到了解决方案。看来问题是csv模块在字节顺序标记（BOM）上被绊倒了。我可以通过encoding='utf-8-sig'打开文件作为建议here来修复它。

python3 URLError未知网址类型为http

2 个答案: