以下代码偶尔出错。如果我只启动一个进程,它工作正常。但是我不断增加进程数,可能是11并且它开始抛出错误。
try:
num_workers = int(sys.argv[1])
except:
num_workers = 1
someval = 10
def do_work(in_queue,x):
i = 0
while True:
item = in_queue.get()
line_no, line = item
# exit signal
if line == None:
if i > 0 :
work.put(i,)
# work.put(i)
return
else:
print "value from work " + line.rstrip('\n')
i = i + 1
if __name__ == "__main__":
manager = Manager()
work = manager.Queue(num_workers)
someval = 20
print " Number of workers is " + str(num_workers)
pool = []
for i in xrange(num_workers):
p = Process(target=do_work, args=(work,someval))
p.start()
pool.append(p)
with open("/home/jay/scripts/a.txt") as f:
iters = itertools.chain(f, (None,)*num_workers)
for num_and_line in enumerate(iters):
work.put(num_and_line)
x = 0
for p in pool:
p.join()
文件/home/jay/scripts/a.txt有10行。
如果我这样做
./x.py 7
Number of workers is 7
value from work 1
value from work 2
value from work 3
value from work 4
value from work 5
value from work 6
value from work 7
value from work 8
value from work 9
value from work 10
x is 0
all done
./x.py 11
Number of workers is 11
value from work 1
value from work 2
value from work 3
value from work 4
value from work 5
value from work 6
value from work 7
value from work 8
value from work 9
value from work 10
Process Process-11:
Traceback (most recent call last):
File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "./x.py", line 18, in do_work
line_no, line = item
TypeError: 'int' object is not iterable
x is 0
all done
答案 0 :(得分:2)
work.put(i,)
中的违规行为do_work
您将int
放入队列中,int
被另一名工作人员读取并解压缩。
我同意dano使用多处理.Pool更容易和更短。
if __name__ == "__main__":
pool = multiprocessing.Pool(num_workers)
with open("/home/jay/scripts/a.txt") as f:
mapped = pool.map(do_work, f)
如果您需要来自工作人员的i
,请将其退回并将其存储在mapped
答案 1 :(得分:2)
问题在于from twilio.rest import TwilioRestClient
account_sid = "{{ authorization sid copied correctly from the site }}" # Your Account SID from www.twilio.com/console
auth_token = "{{ authorization token copied correctly from the site }}" # Your Auth Token from www.twilio.com/console
client = TwilioRestClient(account_sid, auth_token)
message = client.messages.create(body="Hey this is a Test SMS",
to="+919551771607", # Replace with your phone number
from_="+14066234282") # Replace with your Twilio number
print(message.sid)
没有按照您的想法行事。您打算将1元组work.put(1,)
放入队列中,但实际上您只是将(1,)
放入队列中。如果您将该行更改为1
,您将看到您期望的行为。
有一个大的work.put((1,))
值的竞争条件允许您的一个子进程在主进程中的for循环完成加载{{1}之前将num_workers
添加到队列中使用1
sentinel值。使用较小的Queue
值,您可以在任何工作进程将(None,)
添加到队列之前完成for循环。
另外,您是否考虑过使用num_workers
,而是使用1
和multiprocessing.Pool
手动创建Pool
?它会简化你的代码。