如何读取inFile中的标题并将标题写入outFile？ （python 3）

Question

我在文件中使用next来解析数据而不是标题行，所以读取完全避免了第一行。如何在读取时拾取标题（但不解析标题行）然后在写入操作上写入标题？

我想要执行此操作的实际数据集是30列和80k行，所以我试图在一次读取操作中完成此操作。

测试数据：

date, animal, color
3/14/2015, cat, blue
3/24/2015, dog, green

代码：

from dateutil.parser import *
import csv

with open('testin.csv', 'r', encoding='utf-8') as inFile, open('testout.csv', 'w', encoding='utf-8') as outFile:
    exampleReader = csv.reader(inFile)
    next(exampleReader, 1)
    exampleData = list(exampleReader)
    exampleWriter = csv.writer(outFile)
    # print a few to see what it's doing
    print('the list', exampleData)
    for item in exampleData:
        item[0] = str(parse(item[0])) # converting date format for sqlite
        del item[2] # dropping column that is not needed
        print('date corrected', item) 
        exampleWriter.writerow(item)

Answer 1

我会使用pandas来获取如此大量的数据：

import io
import pandas as pd

data = """\
date, animal, color, junk
3/14/2015, cat, blue, aaa
3/24/2015, dog, green, bbb
"""
num_cols = 4
all_cols = set(range(num_cols))
skip_cols = set([2,3])

# replace `io.StringIO(data)` with the CSV filename    
df = pd.read_csv(io.StringIO(data),
                 sep=',',
                 skipinitialspace=True,
                 parse_dates=[0],
                 usecols=(all_cols - skip_cols))
print(df)

# save DF as CSV file
df.to_csv('/path/to/new.csv', index=False)

# save DF to SQLite DB
import sqlalchemy
engine = sqlalchemy.create_engine('sqlite:///my_db.sqlite')
df.to_sql('my_table', engine, if_exists='replace')

示例：

In [150]: data = """\
   .....: date, animal, color, junk
   .....: 3/14/2015, cat, blue, aaa
   .....: 3/24/2015, dog, green, bbb
   .....: """

In [151]: num_cols = 4

In [152]: all_cols = set(range(num_cols))

In [153]: skip_cols = set([2,3])

In [154]: df = pd.read_csv(io.StringIO(data),
   .....:                  sep=',',
   .....:                  skipinitialspace=True,
   .....:                  parse_dates=['date'],
   .....:                  usecols=(all_cols - skip_cols))

In [155]: print(df)
        date animal
0 2015-03-14    cat
1 2015-03-24    dog

Answer 2

在处理输入文件的其余部分之前写入标题：

from dateutil.parser import parse
import csv

with open('testin.csv', 'r', encoding='utf-8') as inFile, open('testout.csv', 'w', encoding='utf-8') as outFile:
    exampleReader = csv.reader(inFile)
    header = next(exampleReader)

    exampleWriter = csv.writer(outFile)
    del header[2]    # drop the column from the header
    exampleWriter.writerow(header)

    for row in exampleReader:
        row[0] = parse(row[0]) # converting date format for sqlite
        del row[2] # dropping column that is not needed
        print('date corrected', row) 
        exampleWriter.writerow(row)

我已经重新安排了一些事情，但是，重点是将标题读入带next()的变量，从标题中删除不需要的列，然后将其写入输出文件。然后处理输入文件的其余部分。

重要的一点是输入文件的其余部分在for循环中逐行处理。当您可以迭代它时，没有必要将整个文件预先读入列表。

您还可以使用生成器表达式来有效地写入行：

from dateutil.parser import parse
import csv

def process_row(row, is_header=False):
    if not is_header:
        row[0] = parse(row[0])
    del row[2]
    return row

with open('data', 'r', encoding='utf-8') as inFile, open('testout.csv', 'w', encoding='utf-8') as outFile:
    exampleReader = csv.reader(inFile)
    header = next(exampleReader)

    exampleWriter = csv.writer(outFile)
    exampleWriter.writerow(process_row(header, is_header=True))

    exampleWriter.writerows(process_row(row) for row in exampleReader)

Answer 3

您可以执行以下操作。我省略了print行以保持简短。

from dateutil.parser import *
import csv

with open('testin.csv', 'r') as inFile, open('testout.csv', 'w') as outFile:
    exampleReader = csv.reader(inFile)
    headers = next(exampleReader)
    exampleWriter = csv.writer(outFile)
    exampleWriter.writerow(headers)
    for item in exampleReader:
        item[0] = str(parse(item[0])) # converting date format for sqlite
        del item[2] # dropping column that is not needed
        exampleWriter.writerow(item)

请勿在生产代码中使用exampleData = list(exampleReader)。它会占用你所有的记忆。