Question

我在一个目录中有16个不同的csv文件，我正在尝试将它们加载到一个pandas数据帧中。每个文件都有datetime和float64列。所有CSV文件都没有列标题。目录

location = os.path.join(base_dir, "DirectoryName")
symbols = os.listdir(location)
df = pd.DataFrame(index=dates)
for symbol in symbols:
    location = os.path.join(base_dir, "DirectoryName", symbol)
    df_temp = pd.read_csv(location, index_col=0, parse_dates=True, dayfirst=True, na_values=['nan'])
    df_temp.dropna()
    df_temp.index = df_temp.index.normalize()
    df_temp = normalize_data(df_temp)
    df = df.join(df_temp)

我现在遇到的问题是最终数据框df，它的索引是datetime，但它的对应行值是列名，第一行是NaN

这是快照

我必须删除df的第一行，但在执行其他操作时这将无济于事，因为某些数据将会丢失。我无法重命名列标题，因为每个文件都不同，我只知道如何静态更改。

Answer 1

我只下载了以下文件：

['hash_rate.csv',
 'difficulty.csv',
 'cost_per_tx.csv',
 'block_size.csv',
 'avg_block_size.csv']

这就是为什么您会在最终的DF中看到相应的数据部分。

请在代码中找到评论。

代码：

import os
import glob
from collections import defaultdict
import pandas as pd

def read_files(filelist):
    # `dfs` - will contain a list of DFs
    # that will be concatenated later on
    dfs = []
    for fn in filelist:
        # parse column name from filename
        col = os.path.splitext(os.path.split(fn)[-1])[0]
        # read individual CSV (as data blocks from defaultdict) into temp DF
        # and add this temporary DF into `dfs` list
        dfs.append(pd.read_csv(
                        fn,
                        parse_dates=[0],
                        header=None,
                        index_col='date',
                        names=['date', col]
                   )
        )
    # return concatenated horizontally (axis=1) DF
    return pd.concat(dfs, axis=1)

def main():
    data_files_mask = r'D:\temp\.data\36827502\*.csv'
    df = read_files(glob.glob(data_files_mask))
    print(df)

if __name__ == '__main__':
    main()

输出：

                     block_size     hash_rate  avg_block_size  cost_per_tx  \
date
2015-01-05 18:15:05     34469.0  3.479099e+08        0.375637     8.185000
2015-01-06 18:15:05     36219.0  3.323940e+08        0.477130     6.598278
2015-01-07 18:15:05     38212.0  3.560892e+08        0.624724     6.232809
2015-01-08 18:15:05     40943.0  4.261981e+08        0.754424     7.113695
2015-01-09 18:15:05     43021.0  4.099610e+08        0.515467     6.199964
2015-01-10 18:15:05     45487.0  4.655484e+08        0.451940     6.821970
2015-01-11 18:15:05     47963.0  4.920513e+08        0.535354     7.958116
2015-01-12 18:15:05     50594.0  6.940933e+08        0.536199     9.415383
2015-02-04 18:15:05     32832.0  3.413843e+08        0.421406     8.054181
2015-02-05 18:15:05     34523.0  3.479099e+08        0.373642     8.958115

                       difficulty
date
2015-01-05 18:15:05  4.761056e+10
2015-01-06 18:15:05  4.880749e+10
2015-01-07 18:15:05  4.940201e+10
2015-01-08 18:15:05  5.227830e+10
2015-01-09 18:15:05  5.425663e+10
2015-01-10 18:15:05  6.081322e+10
2015-01-11 18:15:05  6.225398e+10
2015-01-12 18:15:05  7.272278e+10
2015-02-04 18:15:05  4.671755e+10
2015-02-05 18:15:05  4.761056e+10

Answer 2

考虑使用read_csv's names参数显式定义列，使用循环中的文件名symbol（当然替换.csv扩展名）：

for symbol in symbols:
    ...
    df_temp = pd.read_csv(location, 
                          index_col=0, 
                          parse_dates=True, 
                          dayfirst=True, 
                          na_values=['nan'],
                          header=None,
                          names=['date', symbol.replace('.csv', '')])

Pandas：如何使用文件名水平合并多个CSV（键，值）文件并在结果DF中命名`value`列

2 个答案: