Question

我正在尝试根据一些条件将csv拆分为多个文件。例如，我有一个csv，如下所示：

ID    Timestamp  Product  Price
XX      T1         P1       10  
XX      T2         P1       11
XX      T2         P1       12
XX      T3         P1       13
XX      T3         P1       14
YY      T1         P1       20
YY      T1         P2       25

预期输出：

文件1：XX_P1_file1.csv

ID    Timestamp  Product  Price
XX      T1         P1.      10  
XX      T2         P1.      11
XX      T3         P1       13

文件2：XX_P1_file2.csv

ID    Timestamp  Product  Price
XX      T2         P1       12
XX      T3         P1       14

文件3：YY_P1_file1.csv

ID    Timestamp  Product  Price
YY      T1         P1       20

文件4：YY_P2_file1.csv

ID    Timestamp  Product  Price
YY      T1         P2       25

当前，代码仅查找key（ID，Product），我想在“ Timestamp”周围创建一个条件以获得所需的结果，并且添加它很棘手。代码：

    filein = open(filepath)
    csvin = csv.DictReader(filein)
    csv_files = {}
    files = []
    headers = ['ID','timestamp','product', 'price']

    for row in csvin:
            key = (row['ID'], row['product'])
            if key not in csv_files:
                # create the csv file
                fileout = open('{}_{}.csv'.format(*key), 'w')
                dw = csv.DictWriter(fileout, headers, extrasaction='ignore')
                dw.writeheader()
                csv_files[key] = dw
                files.append(fileout)  # to close them later

            # write the line into to corresponding csv writer
            csv_files[key].writerow(row)

任何帮助将不胜感激。谢谢！

Answer 1

这应该通过使用.cumcount创建一个“文件”列来达到目的。此列稍后将用于帮助动态创建文件名，然后在发送给多个动态命名的文件之前将其删除。 csv文件与.groupby循环出现在两列中，这对于将数据集和随附的文件名分组为动态文件是必需的。由于“时间戳记”重置为1，因此您无需为“产品”列指定任何逻辑，因此将其标记为必须进入新文件。

import pandas as pd
df = pd.read_csv('your_filename.csv')
df['File'] = df.groupby(['ID', 'Timestamp']).cumcount()+1
for (i,f), x in df.groupby(['ID', 'File']):
    x.drop('File', axis=1).to_csv(f'{i}_T{f}_file{f}.csv', index=False)
df

输出：

   ID Timestamp Product  Price  File
0  XX        T1      P1     10     1
1  XX        T2      P1     11     1
3  XX        T3      P1     13     1

   ID Timestamp Product  Price  File
2  XX        T2      P1     12     2
4  XX        T3      P1     14     2

   ID Timestamp Product  Price  File
5  YY        T1      P1     20     1

   ID Timestamp Product  Price  File
6  YY        T1      P2     25     2

Answer 2

这是对您的代码的有效修改。它跟踪ID /产品密钥的实例，以将时间戳定向到正确的文件。假定您的文件已经按sortkey排序（itertools.groupby的要求），但是如果需要，您可以使用csvin=sorted(list(csv.DictReader(filein)),key=sortkey)来预读和排序所有行。

import csv
import itertools
import operator

headers = ['ID', 'Timestamp', 'Product', 'Price']
sortkey = operator.itemgetter('ID', 'Product', 'Timestamp')
files = {}

with open('input.csv', newline='') as filein:
    csvin = csv.DictReader(filein)
    for (id_, product, timestamp), group in itertools.groupby(csvin, key=sortkey):
        for instance, row in enumerate(group, 1):
            key = id_, product, instance
            if key not in files:
                filename = f'{id_}_{product}_file{instance}.csv'
                print(f'Starting {filename}')
                fileout = open(filename, 'w', newline='')
                writer = csv.DictWriter(fileout, headers)
                writer.writeheader()
                files[key] = fileout, writer
            files[key][1].writerow(row)

print(f'Closing {len(files)} output files')
for openfile, _ in files.values():
    openfile.close()

输出：

Starting XX_P1_file1.csv
Starting XX_P1_file2.csv
Starting YY_P1_file1.csv
Starting YY_P2_file1.csv
Closing 4 output files

文件与您输入的内容相符。

将CSV拆分为多个CSV

2 个答案: