我想在multiprocessing中做一些事情，我想推迟获得结果，如下所示：
from multiprocessing import Pool
from twisted.internet import defer
import time
def f(x):
time.sleep(0.5)
print(x)
return x*x
pool = Pool(processes=4) # start 4 worker processes
def get_result(i):
res = pool.apply_async(f, (i, )) # do work in process pool
return defer.Deferred(res.get()) # now, I want to make process do something else, so it should not be blocked
def main():
from twisted.internet import reactor
@defer.inlineCallbacks
def _run():
for i in range(4):
yield get_result(i)
reactor.stop()
reactor.callLater(1, _run)
reactor.run()
if __name__ == '__main__':
main()
答案 0 :(得分：0)
Pool.apply_async() has a mp4 arg that you can leverage to start the callback chain in the socket.emit("load old messages", docs);
socket.on('load old msgs', function(docs) {
for (var i=0; i < docs.length; i++) {
displayMsg(docs[i]);
}
});
.
The catch (which is absolutely crucial to remember) is that the Pool-callback-function will be executed in another thread!
Therefore, you must call socket.on('load old messages', function(docs) {
for (var i=0; i < docs.length; i++) {
displayMsg(docs[i]);
}
});
when applying the result to the callback so that the callback chain occurs in the same thread as the Deferred.
Failure to do this will result in callbacks being executed in a different thread that the reactor has no context into.
Here is a slightly modified example:
reactor.callFromThreadI've printed the thread id so that you can see that Deferred is indeed called in another thread (process?) and not in the main thread, ie reactor thread.
Ommiting reactor in this example will cause the callbacks to executed in a thread that from functools import partial
from multiprocessing import Pool
import threading
import time
from twisted.internet import defer, reactor
def f(x):
time.sleep(5)
return x*x
def get_result(pool, i):
deferred = defer.Deferred() # create a Deferred that will eventually provide a result
_set_result = partial(set_result, deferred=deferred) # pass the Deferred to the apply_async callback
pool.apply_async(f, args=(i,), callback=_set_result) # execute in a separate process, supply callback fn
return deferred
def set_result(result, deferred):
"""
Set the result in the deferred
"""
print('Thread ID: %d, Setting result %d' % (threading.get_ident(), result))
reactor.callFromThread(deferred.callback, result) # execute the Deferred callback chain from the reactor thread
def display_result(result):
"""
Just display the result
"""
print('Thread ID: %d, Display %d' % (threading.get_ident(), result))
def kill_reactor(null):
print('Thread ID: %d, Stopping reactor' % threading.get_ident())
reactor.stop()
def main():
print('Thread ID: %d, Main' % threading.get_ident())
pool = Pool(processes=4)
d = get_result(pool, 3)
d.addCallback(display_result)
d.addCallback(kill_reactor)
reactor.run()
main()
#---------- OUTPUT ----------#
# Thread ID: 803872, Main
# Thread ID: 533632, Setting result 9
# Thread ID: 803872, Display 9
# Thread ID: 803872, Stopping reactor
will not work and Twisted throws up vomit (tracebacks) everywhere!
Consider using at reactor.spawnProcess as this will limit mistakes that you (or myself) would make otherwise.
And as always, if you can do whatever it is you're tyring to do, in a single thread and omit set_result() or reactor.callFromThread(deferred.callback, result), I'd suggest you do that instead.