Question

我有大量的任务要执行，并通过生成器提供结果。但是，使用// Route Source & Destination self.startLOC = CLLocation(latitude: sourceLat, longitude: sourceLong) self.endLOC = CLLocation(latitude: DestinationLat, longitude: DestinationLong) drawPath(startLocation: startLOC, endLocation: endLOC) let marker = GMSMarker() marker.position = CLLocationCoordinate2D(latitude: sourceLat, longitude: sourceLong) // marker.icon = userImage.af_imageScaled(to: CGSize(width: 50, height: 50)).af_imageRoundedIntoCircle() marker.title = "Source" marker.map = mapViewBus let markerr = GMSMarker() markerr.position = CLLocationCoordinate2D(latitude: DestinationLat, longitude: DestinationLong) // markerr.icon = washerImage.af_imageScaled(to: CGSize(width: 50, height: 50)).af_imageRoundedIntoCircle() markerr.title = "Desintation" markerr.map = mapViewBus let camera = GMSCameraPosition.camera(withLatitude: sourceLat, longitude: sourceLong, zoom: 14.0) self.mapViewBus.camera = camera self.mapViewBus.animate(to: camera)和ProcessPoolExecutor将贪婪地评估结果并将它们全部存储在内存中。在生成器中存储了一定数量的结果后，有没有办法阻止？

Answer 1

这样做的想法是将你要处理的内容拆分成块，我将使用与ProcessPoolExecutor documentation中几乎相同的示例：

import concurrent.futures
import math
import itertools as it

PRIMES = [
    293,
    171,
    293,
    773,
    99,
    5419,
    293,
    171,
    293,
    773,
    99,
    5419,
    293,
    171,
    293,
    773,
    99,
    5419]


def is_prime(n):
    if n % 2 == 0:
        return False

    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True


def main():
    with concurrent.futures.ProcessPoolExecutor() as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))

def main_lazy():
    chunks = map(lambda x: it.islice(PRIMES, x, x+4), range(0, len(PRIMES), 4))
    with concurrent.futures.ProcessPoolExecutor() as executor:
        results = zip(PRIMES, 
                      it.chain.from_iterable(map(lambda x: executor.map(is_prime, x), 
                                                 chunks)))
        for number, prime in (next(results) for _ in range(4)):
            print('%d is prime: %s' % (number, prime))

if __name__ == "__main__":
    main_lazy()

请注意main和main_lazy之间的差异，让我们解释一下：

我没有列出我们想要处理的所有内容，而是将其拆分为大小为4的块（使用itertools.islice非常有用），而不是映射< / strong>使用执行程序整个列表我们将映射块。然后只使用python3 lazy map，我们可以将执行器调用懒惰地映射到每个块。所以，我们知道executor.map不是懒惰的，所以当我们请求时会立即评估chunk，但是在我们不要求其他块之前，不会调用executor.map的那些块。。正如您所看到的，我只是从整个结果列表中请求前4个元素，但由于我还使用了itertools.chain，它只会消耗第一个块中的元素，而不计算可迭代的其余部分

所以，既然你想要返回一个生成器，就像从main_lazy函数返回结果一样简单，你甚至可以抽象块大小（可能你需要一个很好的函数来获取propper）块，但这超出了范围）：

def main_lazy(chunk_size): chunks = map(lambda x: it.islice(PRIMES, x, x+chunk_size), range(0, len(PRIMES), chunk_size)) with concurrent.futures.ProcessPoolExecutor() as executor: results = zip(PRIMES, it.chain.from_iterable(map(lambda x: executor.map(is_prime, x), chunks))) return results

Python中的懒惰processpoolexecutor？

1 个答案: