Question

我有一个脚本，它将文本文件作为输入并执行测试。我想要做的是创建两个线程并将输入文本文件分成两部分并运行它们以便最小化执行时间。有没有办法可以做到这一点？

由于

class myThread (threading.Thread):
    def __init__(self, ip_list):
        threading.Thread.__init__(self)
        self.input_list = ip_list

    def run(self):
        # Get lock to synchronize threads
        threadLock.acquire()
        print "python Audit.py " + (",".join(x for x in self.input_list))
        p = subprocess.Popen("python Audit.py " + (",".join(x for x in self.input_list)), shell=True)
        # Free lock to release next thread
        threadLock.release()
        while p.poll() is None:
            print('Test Execution in Progress ....')
            time.sleep(60)

        print('Not sleeping any longer.  Exited with returncode %d' % p.returncode)


def split_list(input_list, split_count):
    for i in range(0, len(input_list), split_count):
        yield input_list[i:i + split_count]

if __name__ == '__main__':

    threadLock = threading.Lock()
    threads = []

    with open("inputList.txt", "r") as Ptr:       
     for i in Ptr:
         try:
             id = str(i).rstrip('\n').rstrip('\r')
             input_list.append(id)
         except Exception as err:
            print err
            print "Exception occured..."
    try:
      test = split_list(input_list, len(input_list)/THREAD_COUNT)
      list_of_lists = list(test)
    except Exception as err:
      print err
      print "Exception caught in splitting list"

    try:
      #Create Threads & Start
      for i in range(0,len(list_of_lists)-1):
         # Create new threads
         threads.append(myThread(list_of_lists[i]))
         threads[i].start()
         time.sleep(1)

      # Wait for all threads to complete
      for thread in threads:
          thread.join()
      print "Exiting Main Thread..!"
    except Exception as err:
      print err
      print "Exception caught during THREADING..."

Answer 1

一些笔记，按随机顺序排列：

在python中，多线程不是解决计算密集型任务的好方法。更好的方法是多处理： Python: what are the differences between the threading and multiprocessing modules?

对于未共享的资源（在您的情况下，每行将由单个进程专门使用），您不需要锁定。更好的方法是地图功能。

def processing_function(line):
    suprocess.call(["python", "Audit.py", line])

with open('file.txt', 'r') as f:
    lines = f.readlines()

to_process = [lines[:len(lines)//2], lines[len(lines)//2:]]    
p = multiprocessing.Pool(2)
results = p.map(processing_func, to_process)

如果计算需要可变的时间量，具体取决于行，使用队列在进程之间移动数据而不是映射可能有助于平衡负载

Answer 2

你正在尝试同时做两件事，这就是并行的定义。这里的问题是，如果你使用CPython，由于GIL（全局解释器锁），你无法做到并行性。 GIL确保只运行一个线程，因为python解释器不被认为是线程安全的。

如果你真的想要并行执行两个操作，你应该使用的是使用多处理模块（导入多处理）

阅读本文：Multiprocessing vs Threading Python

正确的Python线程处理方法

2 个答案: