我使用Python map_async并行下载(使用wget)多个文件,但似乎每次只下载几个文件然后程序就存在了。 顺便说一下,当我将“快速完成”功能传递给map_async时,下面的代码工作正常,所以我想我是否错过了“长时间运行的功能”的任何内容?
这是我的代码
##### main Func calling: map_async
arglist = [ (src, dest, log) for log in DownloadList]
output = MultiFunc(dnldfd, arglist)
### func which used to download file use wget
def dnldfd(arglist):
src,dest,pattern = arglist
if pattern is not None:
# --cut-dirs set "super large" Num
cmd = "wget -q -nc -r -l1 --no-parent --cut-dirs=10 -nH -np -A \"*{pattern}*\" {src} -P {dest}".format_map(vars())
else:
# download all files in http , if no pattern specified
cmd = "wget -q -nc -r -nH -np --cut-dirs=10 -R \"index.html*\" {src} -P {dest}".format_map(vars())
### run "wget" command
os.system(cmd)
### define a common func to call any "func" I want to run in parallel
def MultiFunc(func,arglist):
p = Pool(cpu_count())
result = p.map_async(func, arglist)
try:
out= result.get()
except:
print('result is Empty !!!')
return "Empty result !"
p.close()
p.join() # Here , it should prevent main process quit before sub-process complete , right ?
outlist = []
for o in out:
#outlist.append(json.loads(o))
outlist.append(o)
return outlist