Question

我正在调用函数task(url, param1, param2)，如果API调用不起作用，则会返回API调用url = url或url名称的结果。我的task看起来像是：

def task(url, param1, param2):
    try:
        make_api_call(url, param1, param2)
    except ValueError as e:
        print("val error")            
        return url

现在，我想将task应用于100个网址的列表，并将其multiprocessing作为：

import multiprocessing as mp

def run_tasks(urls, param1, param2):
    jobs = []
    for i in range(len(urls)):
        process = mp.Process(target=task, args=(urls[i], param1, param2))
        jobs.append(process)

    ## catch error processes
    error_urls = []

    ## start processes
    for j in jobs:
        j.start()

    ## finish processes
    for j in jobs:
        j.join()

从上面run_tasks，我如何返回给我url ValueError的{{1}}列表？我试过error_urls.append(j.join())，但这没有用。

Answer 1

有两种方法可以从过程中获得结果。

方法1.使用list中的Manager。您不需要使用锁来在进程之间进行同步。

from multiprocessing import Process, Manager

def task(url, param1, param2, error_list):
    try:
        make_api_call(url, param1, param2)
    except ValueError as e:
        print("val error")            
        error_list.append(url)

def run_tasks(urls, param1, param2):

    error_list = Manager().list()    
    jobs = []

    for i in range(len(urls)):
        process = Process(target=task, args=(urls[i], param1, param2, error_list))
        jobs.append(process)

    ## start processes
    for j in jobs:
        j.start()

    ## finish processes
    for j in jobs:
        j.join()

方法2.使用ProcessPoolExecutor中的concurrent.futures。这种方法易于理解，代码少。

from concurrent import futures

def task(url, param1, param2):
    try:
        make_api_call(url, param1, param2)
    except ValueError as e:
        print("val error")            
        return url

def runt_tasks(urls, param1, param2):

    with futures.ProcessPoolExecutor() as executor:
        result = executor.map(task, urls, [param1] * len(urls), [param2] * len(urls))

    error_list = [item for item in result if item is not None]

最后，从问题的描述。这是一个IO敏感问题。我建议你使用ThreadPoolExecutor。当您执行IO操作时，线程将释放GIL以允许其他线程运行。对于CPU敏感问题，最好使用ProcessPoolExecutor。 asyncio是在Python 3中进行并发编程的另一种选择。

Answer 2

尝试共享内存。使用此public class LMV { static final int NOT_POSSIBLE = -1000000000; // if out put is this i.e soln not possible static int longestSubsequence(String s, char[] c) { //exit conditions if(s.length() ==0 || c.length ==0){ return 0; } if(s.length() < c.length){ return NOT_POSSIBLE; } if(s.length() == c.length){ for(int i=0; i<s.length(); i++){ if(s.charAt(i) !=c [i]){ return NOT_POSSIBLE; } } return s.length(); } if(s.charAt(0) < c[0]){ // ignore, go ahead with next item return longestSubsequence(s.substring(1), c); } else if (s.charAt(0) == c[0]){ // <case 1> include item and start search for next item in chars // <case 2> include but search for same item again in chars // <case 3> don't include item return Math.max( Math.max( ( 1+longestSubsequence(s.substring(1), Arrays.copyOfRange(c, 1, c.length) ) ), ( 1+longestSubsequence(s.substring(1), c ) ) ), ( longestSubsequence(s.substring(1), c )) ); } else { //ignore return longestSubsequence(s.substring(1), c); } } public static void main(String[] args) { char[] chars = {'a', 'e', 'i', 'o', 'u'}; String s1 = "aeio"; String s2 = "aaeeieou"; String s3 = "aaeeeieiioiiouu"; System.out.println(longestSubsequence(s1, chars)); System.out.println(longestSubsequence(s2, chars)); System.out.println(longestSubsequence(s3, chars)); }

您可以在run_tasks中定义此内容

multiprocessing.sharedctypes.Array(typecode_or_type, size_or_initializer, *args[, lock])

并且

from multiprocessing import Process, Lock
from multiprocessing.sharedctypes import Array
lock = Lock()
error_urls = Array(c_char_p, [], lock = lock)

作为Array（）的文档：

与RawArray（）相同，但取决于lock a的值可以返回过程安全的同步包装而不是原始ctypes数组。

因此它是过程安全的。关于Array（）的更多信息可以参考this，关于ctypes（c_char_p）参考this

从多处理中的异常返回值

2 个答案: