Python 3多处理和套接字

时间:2016-07-06 15:45:50

标签: python sockets multiprocessing

我有一个与多个服务器通信的客户端脚本。从我的测试和研究中可以看出,python的套接字库是同步的。无论我做什么,单个套接字变量或动态套接字变量,脚本都不会同时发送和接收数据。这是使用动态套接字ID的脚本的基本版本:

import socket
import logging
import os
import time
from time import time
import datetime
import multiprocessing

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logging.getLogger('requests').setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

IP=[0,1,2,3,4,5]
IP[0] = '192.168.1.21'
IP[1] = '192.168.1.22'
IP[2] = '192.168.1.23'
IP[3] = '192.168.1.24'
IP[4] = '192.168.1.25'
IP[5] = '192.168.1.26'
PORT = 1292

def ClientWorker(ip, port, socketid):
            name = multiprocessing.current_process().name
            ts = time()
            s = [0,1,2,3,4,5]
            s[socketid] = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            s[socketid].connect((ip, port))
            s[socketid].send(b"Get status")
            response = s[socketid].recv(50)
            s[socketid].close()
            print('Connection to {}:'.format(ip))
            print('Process Took: {}'.format(time() - ts))

def main():
    tsmain = time()
    socketnum = 0
    workers = []
    for ipaddr in IP:
        logger.info('Queueing {}'.format(ipaddr))
        worker = multiprocessing.Process(target=ClientWorker, args=(ipaddr, PORT, socketnum))
        workers.append(worker)
        worker.start()
        socketnum += 1
    for w in workers:
        w.join()
    print('Main Took {}'.format(time() - tsmain))

if __name__ == '__main__':
    main()

多处理输出:

2016-07-06 10:32:17,178 - __main__ - INFO - Starting 192.168.1.21
2016-07-06 10:32:17,181 - __main__ - INFO - Starting 192.168.1.22
2016-07-06 10:32:17,181 - __main__ - INFO - Starting 192.168.1.23
2016-07-06 10:32:17,183 - __main__ - INFO - Starting 192.168.1.24
2016-07-06 10:32:17,184 - __main__ - INFO - Starting 192.168.1.25
2016-07-06 10:32:17,185 - __main__ - INFO - Starting 192.168.1.26
Connection to 192.168.1.22: 
Process Took 0.36358118057250977
Connection to 192.168.1.25: 
Process Took 0.37239646911621094
Connection to 192.168.1.21: 
Process Took 0.3772423267364502
Connection to 192.168.1.26: 
Process Took 0.4828777313232422
Connection to 192.168.1.24: 
Process Took 0.6024985313415527
Connection to 192.168.1.23: 
Process Took 0.7020003795623779
Main Took 0.708820104598999

看起来所有进程几乎同时启动,但每个进程需要更长时间才能完成,这表明只能同时打开一个套接字。

我已经对twisted,asyncio& amp;做了研究。 asyncore。我开始沿着扭曲的道路走下去,但由于发送到每个服务器的消息不一样,我无法绕过它。该脚本将被设计为同时向每个服务器发送不同的任务并等待结果。

修改

更详细的输出:

(0, 0) = Process-1 Work started: 2016-07-06 15:42:02.227847
(0, 1) = Process-2 Work started: 2016-07-06 15:42:02.228606
(0, 2) = Process-3 Work started: 2016-07-06 15:42:02.229448
(0, 3) = Process-4 Work started: 2016-07-06 15:42:02.231957
(0, 4) = Process-5 Work started: 2016-07-06 15:42:02.232951
(0, 5) = Process-6 Work started: 2016-07-06 15:42:02.233697
(1, 0) = Process-1 Setting up network socket 0 @ 2016-07-06 15:42:02.227894
(1, 1) = Process-2 Setting up network socket 1 @ 2016-07-06 15:42:02.228655
(1, 2) = Process-3 Setting up network socket 2 @ 2016-07-06 15:42:02.229498
(1, 3) = Process-4 Setting up network socket 3 @ 2016-07-06 15:42:02.232006
(1, 4) = Process-5 Setting up network socket 4 @ 2016-07-06 15:42:02.232994
(1, 5) = Process-6 Setting up network socket 5 @ 2016-07-06 15:42:02.233742
(2, 0) = Process-1 Completed network socket setup 0 @ 2016-07-06 15:42:02.228005 - took 0:00:00.000121
(2, 1) = Process-2 Completed network socket setup 1 @ 2016-07-06 15:42:02.228781 - took 0:00:00.000137
(2, 2) = Process-3 Completed network socket setup 2 @ 2016-07-06 15:42:02.229608 - took 0:00:00.000120
(2, 3) = Process-4 Completed network socket setup 3 @ 2016-07-06 15:42:02.232109 - took 0:00:00.000112
(2, 4) = Process-5 Completed network socket setup 4 @ 2016-07-06 15:42:02.233081 - took 0:00:00.000092
(2, 5) = Process-6 Completed network socket setup 5 @ 2016-07-06 15:42:02.233854 - took 0:00:00.000122
(3, 0) = Process-1 Connecting to network socket @ 2016-07-06 15:42:02.228046
(3, 1) = Process-2 Connecting to network socket @ 2016-07-06 15:42:02.228823
(3, 2) = Process-3 Connecting to network socket @ 2016-07-06 15:42:02.229647
(3, 3) = Process-4 Connecting to network socket @ 2016-07-06 15:42:02.232145
(3, 4) = Process-5 Connecting to network socket @ 2016-07-06 15:42:02.233109
(3, 5) = Process-6 Connecting to network socket @ 2016-07-06 15:42:02.233893
(4, 0) = Process-1 Connected to network socket @ 2016-07-06 15:42:02.232009 - took 0:00:00.003983
(4, 1) = Process-2 Connected to network socket @ 2016-07-06 15:42:02.232624 - took 0:00:00.003818
(4, 2) = Process-3 Connected to network socket @ 2016-07-06 15:42:02.233706 - took 0:00:00.004080
(4, 3) = Process-4 Connected to network socket @ 2016-07-06 15:42:02.235946 - took 0:00:00.003827
(4, 4) = Process-5 Connected to network socket @ 2016-07-06 15:42:02.236215 - took 0:00:00.003120
(4, 5) = Process-6 Connected to network socket @ 2016-07-06 15:42:02.237408 - took 0:00:00.003530
(5, 0) = Process-1 Beginning to send data to network socket @ 2016-07-06 15:42:02.228046
(5, 1) = Process-2 Beginning to send data to network socket @ 2016-07-06 15:42:02.228823
(5, 2) = Process-3 Beginning to send data to network socket @ 2016-07-06 15:42:02.229647
(5, 3) = Process-4 Beginning to send data to network socket @ 2016-07-06 15:42:02.232145
(5, 4) = Process-5 Beginning to send data to network socket @ 2016-07-06 15:42:02.233109
(5, 5) = Process-6 Beginning to send data to network socket @ 2016-07-06 15:42:02.233893
(6, 0) = Process-1 Completed data send to network socket @ 2016-07-06 15:42:02.232084 - took 0:00:00.000038
(6, 1) = Process-2 Completed data send to network socket @ 2016-07-06 15:42:02.232699 - took 0:00:00.000039
(6, 2) = Process-3 Completed data send to network socket @ 2016-07-06 15:42:02.233788 - took 0:00:00.000044
(6, 3) = Process-4 Completed data send to network socket @ 2016-07-06 15:42:02.236024 - took 0:00:00.000035
(6, 4) = Process-5 Completed data send to network socket @ 2016-07-06 15:42:02.236263 - took 0:00:00.000023
(6, 5) = Process-6 Completed data send to network socket @ 2016-07-06 15:42:02.237456 - took 0:00:00.000022
(7, 0) = Process-1 Beginning to wait for data from network socket @ 2016-07-06 15:42:02.232102
(7, 1) = Process-2 Beginning to wait for data from network socket @ 2016-07-06 15:42:02.232717
(7, 2) = Process-3 Beginning to wait for data from network socket @ 2016-07-06 15:42:02.233809
(7, 3) = Process-4 Beginning to wait for data from network socket @ 2016-07-06 15:42:02.236041
(7, 4) = Process-5 Beginning to wait for data from network socket @ 2016-07-06 15:42:02.236274
(7, 5) = Process-6 Beginning to wait for data from network socket @ 2016-07-06 15:42:02.237467
(8, 0) = Process-1 Received data from network socket @ 2016-07-06 15:42:02.273043 - took 0:00:00.040980
(8, 1) = Process-2 Received data from network socket @ 2016-07-06 15:42:02.277420 - took 0:00:00.044757
(8, 2) = Process-3 Received data from network socket @ 2016-07-06 15:42:02.267754 - took 0:00:00.033984
(8, 3) = Process-4 Received data from network socket @ 2016-07-06 15:42:02.281372 - took 0:00:00.045383
(8, 4) = Process-5 Received data from network socket @ 2016-07-06 15:42:02.278337 - took 0:00:00.042100
(8, 5) = Process-6 Received data from network socket @ 2016-07-06 15:42:03.197792 - took 0:00:00.960364
(9, 0) = Process-1 Beginning to process response from server @ 2016-07-06 15:42:02.273101
(9, 1) = Process-2 Beginning to process response from server @ 2016-07-06 15:42:02.277494
(9, 2) = Process-3 Beginning to process response from server @ 2016-07-06 15:42:02.267812
(9, 3) = Process-4 Beginning to process response from server @ 2016-07-06 15:42:02.281444
(9, 4) = Process-5 Beginning to process response from server @ 2016-07-06 15:42:02.278390
(9, 5) = Process-6 Beginning to process response from server @ 2016-07-06 15:42:03.197861
(10, 0) = Process-1 Finished processing response from server @ 2016-07-06 15:42:02.273140 - took 0:00:00.000048
(10, 1) = Process-2 Finished processing response from server @ 2016-07-06 15:42:02.277529 - took 0:00:00.000043
(10, 2) = Process-3 Finished processing response from server @ 2016-07-06 15:42:02.267847 - took 0:00:00.000045
(10, 3) = Process-4 Finished processing response from server @ 2016-07-06 15:42:02.281475 - took 0:00:00.000038
(10, 4) = Process-5 Finished processing response from server @ 2016-07-06 15:42:02.278418 - took 0:00:00.000035
(10, 5) = Process-6 Finished processing response from server @ 2016-07-06 15:42:03.197889 - took 0:00:00.000035
(11, 0) = Process-1 Communicating with server to retreive additional information @ 2016-07-06 15:42:02.273156
(11, 1) = Process-2 Communicating with server to retreive additional information @ 2016-07-06 15:42:02.277544
(11, 2) = Process-3 Communicating with server to retreive additional information @ 2016-07-06 15:42:02.267865
(11, 3) = Process-4 Communicating with server to retreive additional information @ 2016-07-06 15:42:02.281488
(11, 4) = Process-5 Communicating with server to retreive additional information @ 2016-07-06 15:42:02.278432
(11, 5) = Process-6 Communicating with server to retreive additional information @ 2016-07-06 15:42:03.197903
(12, 0) = Process-1 Finished final retreival of data @ 2016-07-06 15:42:02.848586 - took 0:00:00.575468
(12, 1) = Process-2 Finished final retreival of data @ 2016-07-06 15:42:03.637331 - took 0:00:01.359825
(12, 2) = Process-3 Finished final retreival of data @ 2016-07-06 15:42:02.765458 - took 0:00:00.497645
(12, 3) = Process-4 Finished final retreival of data @ 2016-07-06 15:42:03.541048 - took 0:00:01.259597
(12, 4) = Process-5 Finished final retreival of data @ 2016-07-06 15:42:02.818199 - took 0:00:00.539810
(12, 5) = Process-6 Finished final retreival of data @ 2016-07-06 15:42:03.526207 - took 0:00:00.328342
(13, 0) = Process-1 Closing network socket 0 @ 2016-07-06 15:42:02.848643
(13, 1) = Process-2 Closing network socket 1 @ 2016-07-06 15:42:03.637387
(13, 2) = Process-3 Closing network socket 2 @ 2016-07-06 15:42:02.765536
(13, 3) = Process-4 Closing network socket 3 @ 2016-07-06 15:42:03.541101
(13, 4) = Process-5 Closing network socket 4 @ 2016-07-06 15:42:02.818264
(13, 5) = Process-6 Closing network socket 5 @ 2016-07-06 15:42:03.526262
(14, 0) = Process-1 Closed network socket 0 @ 2016-07-06 15:42:02.848719 - took 0:00:00.000086
(14, 1) = Process-2 Closed network socket 1 @ 2016-07-06 15:42:03.637463 - took 0:00:00.000086
(14, 2) = Process-3 Closed network socket 2 @ 2016-07-06 15:42:02.765647 - took 0:00:00.000128
(14, 3) = Process-4 Closed network socket 3 @ 2016-07-06 15:42:03.541168 - took 0:00:00.000075
(14, 4) = Process-5 Closed network socket 4 @ 2016-07-06 15:42:02.818365 - took 0:00:00.000114
(14, 5) = Process-6 Closed network socket 5 @ 2016-07-06 15:42:03.526333 - took 0:00:00.000080
(15, 0) = Process-1 Took 0:00:00.620923
(15, 1) = Process-2 Took 0:00:01.408908
(15, 2) = Process-3 Took 0:00:00.536262
(15, 3) = Process-4 Took 0:00:01.309261
(15, 4) = Process-5 Took 0:00:00.585469
(15, 5) = Process-6 Took 0:00:01.292729
(16, 0) = Process-1 Work finished: 2016-07-06 15:42:02.848763
(16, 1) = Process-2 Work finished: 2016-07-06 15:42:03.637487
(16, 2) = Process-3 Work finished: 2016-07-06 15:42:02.765685
(16, 3) = Process-4 Work finished: 2016-07-06 15:42:03.541191
(16, 4) = Process-5 Work finished: 2016-07-06 15:42:02.818397
(16, 5) = Process-6 Work finished: 2016-07-06 15:42:03.526357
Main Took 0:00:01.430665

1 个答案:

答案 0 :(得分:0)

我确实发现我的原始代码很好,而且时间差异是由于终端机器忙于其他任务或者总体上响应速度慢所致。