Question

我正在读取csv的列表，执行计算，并将输出写入驱动器。数据集很大（16gb RAM上2 gb csv），计算成本很高，输出也很大。因此，我想使用发电机；这样我就可以一次写入一个输出文件。我使用的功能很大，因此不在此处粘贴。我认为带有final for循环的生成器将完成这项工作。

#read csv with indexing, uecols, dtypes ~ 1 DF is approx 50mb in memory
def reader_1sec_csv(filepath,.....):
    return pd.read_csv(filepath,........)

#processing df
def booleans (dataframe):
    dataframe = dataframe[....]
    ....
    ....
    return dataframe

#processing, row by row operation
def activity (row):
    val = "Unknown"
    ...
    ...
    return val

#apply the functions to a list of filepaths and write to outdir
def csv_to_result (filepath, outdir =".."):
    dataframe = booleans(reader_1sec_csv(filepath))
    dataframe["Activity"] = dataframe.apply(activity, axis=1)
    out = dataframe[["Activity"]]
    out.to_csv(os.path.join(outdir, os.path.splitext(os.path.basename(filepath))[0]+'_A.csv'))

#Can I make this into a generator? Keeping the exceptions list?
exceptions = []
for i in filepaths:
    try:
        csv_to_outcsv(i)
    except:
        exceptions.append(i)
        continue

Answer 1

我要附加例外列表。因此，我使用time模块将程序设置为每5分钟运行一次。因为在生成器内部定义所有异常并不容易。

exceptions_list = []
def gen_out(paths):

    for i in paths:
        try:
            yield csv_to_outcsv(i)
        except:
            exceptions_list.append(i)
            continue

def periodic_work(interval):
    while True:
        next(gen_out(filepaths)
        time.sleep(interval)

转换for循环，将不同的函数调用到生成器

1 个答案: