Question

我使用Python。我有100个zip文件。每个zipfile包含100多个xmlfiles。使用xmlfiles我创建了csvfiles。

from xml.etree.ElementTree import fromstring
import zipfile
from multiprocessing import Process

def parse_xml_for_csv1(data, writer1):
    root = fromstring(data)
    for node in root.iter('name'):
        writer1.writerow(node.get('value'))

def create_csv1():
    with open('output1.csv', 'w') as f1:
        writer1 = csv.writer(f1)

        for i in range(1, 100):
            z = zipfile.ZipFile('xml' + str(i) + '.zip')
            # z.namelist() contains more than 100 xml files
            for finfo in z.namelist():
                data = z.read(finfo) 
                parse_xml_for_csv1(data, writer1)


def create_csv2():
    with open('output2.csv', 'w') as f2:
        writer2 = csv.writer(f2)

        for i in range(1, 100): 
            ...


if __name__ == "__main__":
    p1 = Process(target=create_csv1)
    p2 = Process(target=create_csv2)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

请告诉我，如何优化我的代码？让代码更快？

Answer 1

您只需要使用参数定义一个方法。在给定数量的线程或进程中拆分100个.zip文件的处理。您将添加的进程越多，您将使用的CPU越多，也许您可以使用2个以上的进程，它会更快（由于某些时候磁盘I / O可能存在瓶颈）

在下面的代码中，我可以更改为4或10个进程，无需复制/粘贴代码。它处理不同的zip文件。

你的代码并行处理相同的100个文件：它甚至比没有多处理的情况要慢！

def create_csv(start_index,step):
    with open('output{0}.csv'.format(start_index//step), 'w') as f1:
        writer1 = csv.writer(f1)

        for i in range(start_index, start_index+step):
            z = zipfile.ZipFile('xml' + str(i) + '.zip')
            # z.namelist() contains more than 100 xml files
            for finfo in z.namelist():
                data = z.read(finfo)
                parse_xml_for_csv1(data, writer1)



if __name__ == "__main__":
    nb_files = 100
    nb_processes = 2   # raise to 4 or 8 depending on your machine

    step = nb_files//nb_processes
    lp = []
    for start_index in range(1,nb_files,step):
        p = Process(target=create_csv,args=[start_index,step])
        p.start()
        lp.append(p)
    for p in lp:
        p.join()

Python，多处理：如何优化代码？让代码更快？

1 个答案: