Python map_async并行“wget文件” - 但下载总是在中间中止,只有少量文件完成下载

时间:2018-06-12 14:09:02

标签: python-3.x python-multiprocessing

我使用Python map_async并行下载(使用wget)多个文件,但似乎每次只下载几个文件然后程序就存在了。 顺便说一下,当我将“快速完成”功能传递给map_async时,下面的代码工作正常,所以我想我是否错过了“长时间运行的功能”的任何内容?

这是我的代码

##### main Func calling:  map_async
arglist = [ (src, dest, log) for log in DownloadList]
output = MultiFunc(dnldfd, arglist)

### func which used to download file use wget

def dnldfd(arglist):
    src,dest,pattern = arglist

    if pattern is not None:
        # --cut-dirs set "super large" Num
        cmd = "wget -q -nc -r -l1 --no-parent --cut-dirs=10 -nH -np -A \"*{pattern}*\" {src} -P {dest}".format_map(vars())
    else:
        # download all files in http , if no pattern specified  
        cmd = "wget -q -nc -r -nH -np --cut-dirs=10 -R \"index.html*\" {src} -P {dest}".format_map(vars())

    ### run "wget" command
    os.system(cmd) 

### define a common func to call any "func" I want to run in parallel 
def MultiFunc(func,arglist):
    p = Pool(cpu_count())

    result = p.map_async(func, arglist)

    try:
        out= result.get()
    except:
        print('result is Empty !!!')
        return "Empty result !"
    p.close()
    p.join() # Here , it should prevent main process quit before sub-process complete , right ?

    outlist = []
    for o in out:
        #outlist.append(json.loads(o))
        outlist.append(o)

    return outlist 

0 个答案:

没有答案