在python中使用pandas加载csv文件

时间:2018-01-14 05:12:43

标签: python pandas

以下是我的示例数据:

2017-11-27T00:29:37.698-06:00,,"42,00,00,00,3E,51,1B,D7,42,1C,00,00,40"
2017-11-27T00:29:37.698-06:00,,"42,00,00,00,3E,51,1B,D7,42,1C,00,00,40"
2017-11-27T00:29:37.698-06:00,,"42,00,00,00,3E,51,1B,D7,42,1C,00,00,40"

我尝试使用以下方法使用pandas加载数据:

data = pd.read_csv("sample.csv",header = None)

我的输出是:

                0                 1           2
0  2017-11-27T00:29:37.698-06:00 NaN  42,00,00,00,3E,51,1B,D7,42,1C,00,00,40
1  2017-11-27T00:29:37.698-06:00 NaN  42,00,00,00,3E,51,1B,D7,42,1C,00,00,40
2  2017-11-27T00:29:37.698-06:00 NaN  42,00,00,00,3E,51,1B,D7,42,1C,00,00,40

我想将第二列中的每个数据与第一列分开作为时间戳。

我的预期输出是:

    0                             1  2  3  4....
0  2017-11-27T00:29:37.698-06:00  42 00 00 00
1  2017-11-27T00:29:37.698-06:00  42 00 00 00
2  2017-11-27T00:29:37.698-06:00  42 00 00 00

3 个答案:

答案 0 :(得分:3)

使用正则表达式传递sep参数。然后,对数据进行一些清理。

df = pd.read_csv(
      'file.csv', 
      sep='"*,',           # separator
      header=None,         # no headers
      engine='python',     # allows a regex with multiple characters
      index_col=[0]        # specify timestamp as the index
)   

df.iloc[:, 1] = df.iloc[:, 1].str.strip('"').astype(int)
df.iloc[:, -1] = df.iloc[:, -1].str.strip('"').astype(int)

df

                               1   2   3   4   5   6   7   8   9   10  11  12  \
0                                                                               
2017-11-27T00:29:37.698-06:00 NaN  42   0   0   0  3E  51  1B  D7  42  1C   0   
2017-11-27T00:29:37.698-06:00 NaN  42   0   0   0  3E  51  1B  D7  42  1C   0   
2017-11-27T00:29:37.698-06:00 NaN  42   0   0   0  3E  51  1B  D7  42  1C   0   

                               13  14  
0                                      
2017-11-27T00:29:37.698-06:00   0  40  
2017-11-27T00:29:37.698-06:00   0  40  
2017-11-27T00:29:37.698-06:00   0  40  

要使用NaN删除列,请使用dropna -

df.dropna(how='all', axis=1, inplace=True)

答案 1 :(得分:3)

首先将参数parse_dates=[0]添加到解析第一列到日期时间。

然后join到原始split ed列2并删除列12,最后rename所有列都添加{{ 1}}:

1

<强>详细

df = pd.read_csv("sample.csv",header = None, parse_dates=[0])

df = (df.drop([1,2], axis=1)
        .join(df[2].str.split(',', expand=True)
        .rename(columns = lambda x: x+1))   
      )  
print (df)
                       0   1   2   3   4   5   6   7   8   9   10  11  12  13
0 2017-11-27 06:29:37.698  42  00  00  00  3E  51  1B  D7  42  1C  00  00  40
1 2017-11-27 06:29:37.698  42  00  00  00  3E  51  1B  D7  42  1C  00  00  40
2 2017-11-27 06:29:37.698  42  00  00  00  3E  51  1B  D7  42  1C  00  00  40

答案 2 :(得分:3)

如果需要,您可以执行自己的csv解析器,如:

代码:

def read_my_csv(filename):
    with open(filename, 'rU') as f:

        # build csv reader
        reader = csv.reader(f)

        # for each row, check for footer
        for row in reader:
            yield [row[0]] + row[2].split(',')

测试代码:

import csv
import pandas as pd

df = pd.DataFrame(read_my_csv('csvfile.csv'))
print(df)

结果:

                                  0   1   2   3   4   5   6   7   8   9   10  \
0      2017-11-27T00:29:37.698-06:00  42  00  00  00  3E  51  1B  D7  42  1C   
1      2017-11-27T00:29:37.698-06:00  42  00  00  00  3E  51  1B  D7  42  1C   
2      2017-11-27T00:29:37.698-06:00  42  00  00  00  3E  51  1B  D7  42  1C   

   11  12  13  
0  00  00  40  
1  00  00  40  
2  00  00  40