Python多处理池内存问题

时间:2014-04-07 21:34:22

标签: python memory

我想运行一个python进程,如果更新了一些XML,它在后台监视中始终处于活动状态。但是,一段时间后我得到了python [defunct]。我怀疑这是与内存相关的问题,因为我可以看到子进程的内存不断增长。但我不知道如何解决这个问题。这是代码:

def getXmlFile(url):    
    req = urllib2.Request(url)
    rep = urllib2.urlopen(req)
    response_content = rep.read()
    last_modified = rep.info().getheader('Last-Modified')

    return (response_content, last_modified)

def writeXmlFile(content, fileName):
    file_path = os.path.join(OUTPUT, fileName)
    f = open(file_path, 'w')
    f.write(content)    
    f.close()

def downloadXml(xml):
    url = BASE_URL + xml
    (response, last_modified) = getXmlFile(url)

    if LAST_MODIFIED.get(xml) is None or LAST_MODIFIED[xml] != last_modified:
            LAST_MODIFIED[xml] = last_modified
            utc_datetime = datetime.datetime.utcnow()
            currentTime = utc_datetime.strftime('%Y%m%d%H%M%S')
            fileName = xml.replace('.XML', '_%s.XML' % currentTime)
            print "Writing " + fileName
            writeXmlFile(response, fileName)

def main():
    pool = Pool(processes=1)

    while (True):
        pool.map_async(downloadXml, XmlList)
        time.sleep(2)

    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

最初一切都很好。它每2秒监视一定的xmls,如果有更新,它将下载xml。但它会在一段时间后挂起。 Ps aux将使用python [defunc]显示子python进程。

0 个答案:

没有答案