Question

问题陈述

如何使用大块之间重叠的熊猫来大块读取csv文件？

例如，假设列表indexes代表我希望读取的某些数据框的索引。

indexes = [0,1,2,3,4,5,6,7,8,9]

read_csv（文件名，chunksize =无）：

indexes = [0,1,2,3,4,5,6,7,8,9]  # read in all indexes at once

read_csv（文件名，chunksize = 5）：

indexes = [[0,1,2,3,4], [5,6,7,8,9]]  # iteratively read in mutually exclusive index sets

read_csv（文件名，chunksize = 5，重叠= 2 ）：

indexes = [[0,1,2,3,4], [3,4,5,6,7], [6,7,8,9]]  # iteratively read in indexes sets with overlap size 2

工作解决方案

我有一个使用 skiprows 和 nrows 的黑客解决方案，但是随着它读取csv文件，它变得越来越慢。

indexes = [*range(10)]
chunksize = 5
overlap_count = 2
row_count = len(indexes)  # this I can work out before reading the whole file in rather cheaply

chunked_indexes = [(i, i + chunksize) for i in range(0, row_count, chunksize - overlap_count)]  # final chunk here may be janky, assume it works for now (it's more about the logic)
for chunk in chunked_indexes:
    skiprows = [*range(chunk[0], chunk[1])]
    pd.read_csv(filename, skiprows=skiprows, nrows=chunksize)

有人对此问题有见识或改进的解决方案吗？

Answer 1

我认为您应该将数字而不是列表传递给skiprow，请尝试：

for i in list(range(0, row_count-overlap_count, chunksize - overlap_count)):
    print (pd.read_csv('test.csv', 
                       skiprows=i+1, #here it is +1 because the first row was header 
                       nrows=chunksize, 
                       index_col=0, # this was how I save my csv
                       header=None) # you may need to read header before
             .index)
Int64Index([0, 1, 2, 3, 4], dtype='int64', name=0)
Int64Index([3, 4, 5, 6, 7], dtype='int64', name=0)
Int64Index([6, 7, 8, 9], dtype='int64', name=0)

熊猫-块之间有重叠的块read_csv

1 个答案: