我在多个子目录中有一个170,000多个pickle文件的目录,最初使用(protocol = 0)格式进行pickle。这在时间或空间上都不是非常有效。
我写了一个脚本来重新挑选(使用cPickle,protocol = 2)文件夹中的每个文件但奇怪的是,脚本在处理特定文件时抛出异常(文件#95,000)。 最初,我认为pickle文件已损坏。当我尝试从IPython命令行加载这个精确的pickle文件时,文件加载就好了。
所以,我对于为什么会发生这种情况感到茫然。这是我的剧本,我很感激帮助:
import os
import cPickle
import numpy
import time
import re
from progressbar import ProgressBar
inpath = '/path/to/folder'
def list_files(dir):
r = []
subdirs = [x[0] for x in os.walk(dir)]
for subdir in subdirs:
files = os.walk(subdir).next()[2]
if (len(files) > 0):
for file in files:
r.append(subdir + "/" + file)
return r
infileList = list_files(inpath)
print "Total number of files found: %d" % len(infileList)
print "\n\n"
progress = ProgressBar()
outfilename = " "
print "Processing pickle files. Pls wait..."
t0 = time.time()
filecount = 0
for file in progress(infileList):
try:
arr = cPickle.load(open(file , "rb" ))
outfilename = re.sub('/initial/path/','/new/path/',file)
if not os.path.exists(os.path.dirname(outfilename)):
os.makedirs(os.path.dirname(outfilename))
with open(outfilename, "wb") as f:
cPickle.dump(arr,f,protocol=2)
filecount = filecount + 1
except Exception,e:
print "\n" + str(filecount)
print "\nError occured while processing file: " + outfilename
tx = time.time()
print "\n Time elapsed: %.2f" % (tx-t0)
continue
t1 = time.time()
total = t1-t0
print "Files repickled with protocol=2.\nRepickling execution time: %.2f sec" % total