Question

通过查看不完全适合我的案例的例子，我是一个下雪的人......或许他们这样做。因此，如果有很好的例子，我无法用大约3周的Python经验来解释它们

我有一个查询数据库的脚本，收集可下载电影的列表，然后将它们逐个下载到您选择的目录中。我想让它一次下载4或5，因为这需要一个年龄。

这是我试图做的简化版本，myapp是我的数据库应用程序。然而它似乎只是顺序运行，即使它说它正在启动两个线程

listOfID是某些容器的ID，可能有也可能没有电影，然后versionS返回电影文件名。

import threading
import myapp_api

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)
for ID in listOfIDs:
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])

ipath = ('/Users/me/Desktop/scripts/downloads/')

exitFlag = 0

class myThread (threading.Thread):
    def __init__(self, threadID, name, counter):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.counter = counter
    def run(self):
        print "Starting " + self.name
        for ID in listOfIDs:
            print "\nID= " + str(ID) + "\n"
            downLoad(ID)
        print "Exiting " + self.name

def downLoad(ID):
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)
    print "Thread Name = " + threadName


# Create new threads
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread(2, "Thread-2", 2)


# Start new Threads
thread1.start()
thread2.start()

print "Exiting Main Thread"

好的，所以我修改了代码以接受ShadowRanger的建议，它仍然只是一次下载一个，我把它塞进某处？ ...代码现在看起来像这样。

import threading
import myapp_api
from collections import deque

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)
for ID in listOfIDs:
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])

ipath = ('/Users/me/Desktop/scripts/downloads/')

def downLoad(ID):
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)

with closing(multiprocessing.Pool(4)) as pool:
    deque(pool.imap_unordered(downLoad, listOfIDs), maxlen=0)

最后，ShadowRanger的所有建议都是正确的，错误是我做错了（我想我早期迭代了listOfIDs，只将最后一个传递给函数） ......这是最终的工作版本。

   import threading
    import myapp_api
    from collections import deque

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)

ipath = ('/Users/me/Desktop/scripts/downloads/')

def downLoad(ID):
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)


with closing(multiprocessing.Pool(4)) as pool:
    deque(pool.imap_unordered(downLoad, listOfIDs), maxlen=0)

Answer 1

我没看到线程如何分裂工作。看起来他们都下载了同样的东西。

如果目标是根据已知ID下载一堆文件，multiprocessing有一个.dummy模块，可以像multiprocessing一样使用线程实现，这样可以轻松实现线程池：

import multiprocessing.dummy as multiprocessing
from contextlib import closing

with closing(multiprocessing.Pool(4)) as pool: # Pick your favorite number of workers
    pool.map(downLoad, listOfIDs)

Python线程帮助需要

1 个答案: