Question

我有以下代码，当我运行它时，会停顿很长时间，然后在解析后打印一堆行，然后重复该过程。因此，我想知道为什么会有一个暂停，以及如何重新编码以提高内存和速度方面的效率？

#!/usr/bin/env python

import os
from lxml import etree

subdirectories = []

for root, dirs, files in os.walk("/dev/shm/heatmap/OpenHPC"):
    for i in dirs:
        if "compute" in i:
            subdirectories.append(i)

#for i in subdirectories:
#    os.system("cd /dev/shm/heatmap/OpenHPC/" + i + " && rrdtool dump cpu_temp.rrd cpu_temp.xml")

for i in subdirectories:
    with open(i+"/cpu_temp.xml") as f:
        doc = etree.parse(f)
    it = iter(doc.xpath(
        '//comment()[following-sibling::row] | //row/v/text()'
    ))
    for db_date, db_value in zip(it, it):
        with open("temperatureData.txt", 'a+') as f:
            f.write(i + " " + db_date.text.strip()[0:23] + " " +  db_value + "\n")
            print(f.write(i + " " + db_date.text.strip()[0:23] + " " +  db_value + "\n")

Answer 1

文件操作很昂贵，您执行的次数越少越好。我建议一次打开文件以附加和最小化写入。这取决于您有多少内存可供使用，但是如果您有足够的内存，我建议构建一个条目数组并将其批量写入文件一次（为清楚起见，对某些变量进行了重命名）：

#!/usr/bin/env python

import os
from lxml import etree

subdirectories = []

for root, dirs, files in os.walk("/dev/shm/heatmap/OpenHPC"):
    for subdir in dirs:
        if "compute" in subdir:
            subdirectories.append(subdir)

with open('temperatureData.txt', 'a+') as temperatureData:
    datalines = []
    for subdir in subdirectories:
        with open(subdir+"/cpu_temp.xml") as f:
            doc = etree.parse(f)
        it = iter(doc.xpath(
            '//comment()[following-sibling::row] | //row/v/text()'
        ))
        for db_date, db_value in zip(it, it):
            dataline = subdir + " " + db_data.text.strip()[0:23] + " " + db_value
            datalines.append(dataline)
            print(dataline)
    temperatureData.write('\n'.join(datalines))

可以将以下代码更改为更快

1 个答案: