python多处理TypeError:'int'对象不可迭代

时间:2016-05-17 17:06:40

标签: python multiprocessing

以下代码偶尔出错。如果我只启动一个进程,它工作正常。但是我不断增加进程数,可能是11并且它开始抛出错误。

try:
    num_workers = int(sys.argv[1])
except:
    num_workers = 1

someval = 10
def do_work(in_queue,x):
    i = 0
    while True:
        item = in_queue.get()
        line_no, line = item

        # exit signal
        if line == None:
            if i > 0 :
                work.put(i,)
            # work.put(i)
            return
        else:
            print "value from work " + line.rstrip('\n')
            i = i + 1

if __name__ == "__main__":

    manager = Manager()
    work = manager.Queue(num_workers)
    someval = 20
    print " Number of workers is " + str(num_workers)
    pool = []
    for i in xrange(num_workers):
        p = Process(target=do_work, args=(work,someval))
        p.start()
        pool.append(p)
    with open("/home/jay/scripts/a.txt") as f:
         iters = itertools.chain(f, (None,)*num_workers)
         for num_and_line in enumerate(iters):
                work.put(num_and_line)

    x = 0
    for p in pool:
        p.join()

文件/home/jay/scripts/a.txt有10行。

如果我这样做

 ./x.py 7
     Number of workers is 7
    value from work 1
    value from work 2
    value from work 3
    value from work 4
    value from work 5
    value from work 6
    value from work 7
    value from work 8
    value from work 9
    value from work 10
     x is 0
     all done 

./x.py 11
 Number of workers is 11
value from work 1
value from work 2
value from work 3
value from work 4
value from work 5
value from work 6
value from work 7
value from work 8
value from work 9
value from work 10
Process Process-11:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "./x.py", line 18, in do_work
    line_no, line = item
TypeError: 'int' object is not iterable
 x is 0
 all done 

2 个答案:

答案 0 :(得分:2)

work.put(i,)中的违规行为do_work 您将int放入队列中,int被另一名工作人员读取并解压缩。

我同意dano使用多处理.Pool更容易和更短。

if __name__ == "__main__":
    pool = multiprocessing.Pool(num_workers)
    with open("/home/jay/scripts/a.txt") as f:
        mapped = pool.map(do_work, f)

如果您需要来自工作人员的i,请将其退回并将其存储在mapped

答案 1 :(得分:2)

问题在于from twilio.rest import TwilioRestClient account_sid = "{{ authorization sid copied correctly from the site }}" # Your Account SID from www.twilio.com/console auth_token = "{{ authorization token copied correctly from the site }}" # Your Auth Token from www.twilio.com/console client = TwilioRestClient(account_sid, auth_token) message = client.messages.create(body="Hey this is a Test SMS", to="+919551771607", # Replace with your phone number from_="+14066234282") # Replace with your Twilio number print(message.sid) 没有按照您的想法行事。您打算将1元组work.put(1,)放入队列中,但实际上您只是将(1,)放入队列中。如果您将该行更改为1,您将看到您期望的行为。

有一个大的work.put((1,))值的竞争条件允许您的一个子进程在主进程中的for循环完成加载{{1}之前将num_workers添加到队列中使用1 sentinel值。使用较小的Queue值,您可以在任何工作进程将(None,)添加到队列之前完成for循环。

另外,您是否考虑过使用num_workers,而是使用1multiprocessing.Pool手动创建Pool?它会简化你的代码。