Question

我正在编写一个脚本来从网页上下载图像，并且正在尝试使其成为多线程的，因此它要快得多。

在下载功能中，我必须设置两个参数，因为设置一个（队列）时会出现此错误：

TypeError: downloading() takes 1 positional arguments but 21* were given

** queue has 21 links

代码：

count = 0
queue = {"some urls", , , }
done = set()
path = 'foldername'


def downloading(queue, name):
    for imgs in queue:
        if imgs not in done:
            done.add(imgs)
            urllib.request.urlretrieve(imgs, path + '/' + imgs.split('/')[-1])
            global count
            count += 1
            print(str(count) + ' ' + name)
            print('Done:  ' + imgs.split('/')[-1])


def threads(queue):
    print('Start Downloading ...')
    th1 = Thread(target=downloading, args=(queue, "Thread 1"))
    th1.start()
    th2 = Thread(target=downloading, args=(queue, "Thread 2"))
    th2.start()
    th3 = Thread(target=downloading, args=(queue, "Thread 3"))
    th3.start()
    th4 = Thread(target=downloading, args=(queue, "Thread 4"))
    th4.start()
    th5 = Thread(target=downloading, args=(queue, "Thread 5"))
    th5.start()
    th6 = Thread(target=downloading, args=(queue, "Thread 6"))
    th6.start()
    th7 = Thread(target=downloading, args=(queue, "Thread 7"))
    th7.start()
    th8 = Thread(target=downloading, args=(queue, "Thread 8"))
    th8.start()
    th9 = Thread(target=downloading, args=(queue, "Thread 9"))
    th9.start()
    th10 = Thread(target=downloading, args=(queue, "Thread 10"))
    th10.start()

Answer 1

对所需的线程数使用简单的for循环。当然，您应该以某种方式保存它们，以使其最终join：

def threads(queue):
    num_of_threads = 10
    print('Start Downloading ...')
    threads = []
    for i in range(1, num_of_threads+1):
        th = Thread(target=downloading, args=(queue, "Thread {}".format(i)))
        threads.append(th)
        th.start()
    return threads

根据建议，pool在此用例中将有所帮助，因为它将优化对系统的运行：

from multiprocessing.dummy import Pool as ThreadPool

def downloading(img):
    urllib.request.urlretrieve(img, path + '/' + img.split('/')[-1])
    global count
    count += 1
    print('Done:  ' + img.split('/')[-1])

def threads(queue):
    pool = ThreadPool()
    pool.map(downloading, queue)
    pool.close()
    pool.join()

请注意，通过这种方式，您应该更改downloading函数以接收一个作为单个图像的参数。 map函数将可迭代项（第二个参数）的每个项目发送到函数（第一个参数）。这也是为什么done设置不是必需的原因，因为每个图像将被精确地处理一次。

如何在python 3中创建和启动任意数量的线程

1 个答案: