Question

我正在python 3.7中使用多进程池映射而且我在使用docker容器时遇到了如下错误

  multiprocessing.pool.MaybeEncoding Error: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f8ff58e8110>'. Reason: 'PicklingError("Can't pickle <class 'MemoryError'>: it's not the same object as builtins.MemoryError")'

我正在使用如下代码，


def foo(track):

    track_url = track[1]
    url = requests.get(track_url)
    hashes = fingerprint.fingerprint_worker(
        BytesIO(url.content)
    )
    return True

def worker:
 tracks = [[1, "track_url_1"], [2, "track_url_2"]]

 with multiprocessing.Pool(const.MULTIPROCESSING_POOL_SIZE) as pool:
                pool.map(foo, tracks)
                pool.close()
                pool.terminate()
                pool.join()

MULTIPROCESSING_POOL_SIZE = 1

此错误的完整堆栈跟踪为


Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1836, in __call__
 return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in wsgi_app
 response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1403, in handle_exception
  reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise
 raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1817, in wsgi_app
 response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
 rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise 
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/radio-streaming-fingerprint/app.py", line 37, in fingerprint_covers_tracks
resp = fp.fingerprint_tracks(track, generate_fingerprints_with_pool)
File "/radio-streaming-fingerprint/fingerprint.py", line 37, in fingerprint_tracks
pool.map(generate_fingerprints_with_pool, tracks)
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f8ff58e8110>'. Reason: 'PicklingError("Can't pickle <class 'MemoryError'>: it's not the same object as builtins.MemoryError")'

我不明白为什么在生产环境中而不是在本地项目设置中会出现此错误，docker容器中有4GB RAM。

请给我建议解决方法。

谢谢。

无法腌制<class'MemoryError'>：它与内建对象不同

0 个答案: