Question

我正在尝试对Twitter搜索和流API进行多处理，但我无法启动任何流程。我尝试使用更简单的函数，start（）方法也可以。

如果你能告诉我如何解决这些问题，我将非常感激！

谢谢！

此致特伦斯

以下是启动流程的代码的一部分：

    queue = Queue()
    exitEvent = Event()
    multiprocessing.log_to_stderr(logging.DEBUG)

    print "-- starting tweetParser! --"
    tweetParser = Process(target = parseTweets, args = (conn, cursor, queue, exitEvent, config['debug'],))
    tweetParser.daemon = True
    tweetParser.start()

    print "-- starting twitterStream! --"
    twitterStream = Process(target = StreamingTwitter, args = (streamAPI, queue, exitEvent, config['keywords'], config['time'], config['debug'],))
    twitterStream.daemon = True
    twitterStream.start()

    print "-- starting twitterSearch! --"
    twitterSearch = Process(target = SearchingTwitter, args = (searchAPI, queue, exitEvent, config['keywords'], config['time'], config['debug'],))
    twitterSearch.start()

    tweetParser.join()

以下是我的错误的完整追溯：

Traceback (most recent call last):
  File "C:\Python27\Code\tweets\Stream.py", line 620, in <module>
    tweetParser.start()
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 425, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 568, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 425, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 687, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 655, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 686, in _batch_setitems
    save(k)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 754, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <type 'NoneType'>: it's not found as __builtin__.NoneType
[INFO/MainProcess] process shutting down
[DEBUG/MainProcess] running all "atexit" finalizers with priority >= 0
[DEBUG/MainProcess] running the remaining "atexit" finalizers
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 864, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 886, in load_eof
    raise EOFError
EOFError
[INFO/Process-1] process shutting down
[DEBUG/Process-1] running all "atexit" finalizers with priority >= 0
[DEBUG/Process-1] running the remaining "atexit" finalizers

Answer 1

您遇到了序列化问题。您无法使用None腌制multiprocessing。但是你可以使用multiprocess。它是一个使用更好的序列化的分支。

>>> import multiprocess
>>> multiprocess.Pool().map(lambda x: x, [None, None, None])
[None, None, None]
>>>

这可能就是你所需要的。

多处理流和搜索API导致PickleError：NoneType

1 个答案: