Question

我是Python的新手，我正在为我正在参加的数据科学课程开展一个项目。我有一个很大的csv文件（大约1.9亿行，大约7GB的数据），我首先需要做一些数据准备。

完整免责声明：此处的数据来自此Kaggle competition。

来自Jupyter Notebook的图片随后有标题。虽然它读取full_data.head()，但我使用的是100,000行样本来测试代码。

最重要的一栏是click_time。格式为：dd hh:mm:ss。我想将它分成4个不同的列：日，小时，分钟和秒。我已经达成了一个适用于这个小文件的解决方案，但运行10％的实际数据需要很长时间，更不用说在100％的真实数据上运行了（因为只是阅读了完整的csv现在是个大问题。）

这是：

# First I need to split the values
click = full_data['click_time']
del full_data['click_time']
click = click.str.replace(' ', ':')
click = click.str.split(':')

# Then I transform everything into integers. The last piece of code
# returns an array of lists, one for each line, and each list has 4
# elements. I couldn't figure out another way of making this conversion
click = click.apply(lambda x: list(map(int, x)))

# Now I transform everything into unidimensional arrays
day = np.zeros(len(click), dtype = 'uint8')
hour = np.zeros(len(click), dtype = 'uint8')
minute = np.zeros(len(click), dtype = 'uint8')
second = np.zeros(len(click), dtype = 'uint8')
for i in range(0, len(click)):
    day[i] = click[i][0]
    hour[i] = click[i][1]
    minute[i] = click[i][2]
    second[i] = click[i][3]
del click

# Transforming everything to a Pandas series
day = pd.Series(day, index = full_data.index, dtype = 'uint8')
hour = pd.Series(hour, index = full_data.index, dtype = 'uint8')
minute = pd.Series(minute, index = full_data.index, dtype = 'uint8')
second = pd.Series(second, index = full_data.index, dtype = 'uint8')

# Adding to data frame
full_data['day'] = day
del day
full_data['hour'] = hour
del hour
full_data['minute'] = minute
del minute
full_data['second'] = second
del second

结果还可以，这就是我想要的，但必须有一个更快的方法：

有关如何改进此实施的任何想法？如果对数据集感兴趣，则来自test_sample.csv：https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/data

提前多多感谢!!

编辑1 ：在@COLDSPEED请求后，我提供full_data.head.to_dict()的结果：

  {'app': {0: 12, 1: 25, 2: 12, 3: 13, 4: 12},
  'channel': {0: 497, 1: 259, 2: 212, 3: 477, 4: 178},
  'click_time': {0: '07 09:30:38',
  1: '07 13:40:27',
  2: '07 18:05:24',
  3: '07 04:58:08',
  4: '09 09:00:09'},
  'device': {0: 1, 1: 1, 2: 1, 3: 1, 4: 1},
  'ip': {0: 87540, 1: 105560, 2: 101424, 3: 94584, 4: 68413},
  'is_attributed': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0},
  'os': {0: 13, 1: 17, 2: 19, 3: 13, 4: 1}}

Answer 1

转换为timedelta并提取组件：

v = df.click_time.str.split()

df['days'] = v.str[0].astype(int)
df[['hours', 'minutes', 'seconds']] = (
      pd.to_timedelta(v.str[-1]).dt.components.iloc[:, 1:4]
)

df
   app  channel   click_time  device      ip  is_attributed  os  days  hours  \
0   12      497  07 09:30:38       1   87540              0  13     7      9   
1   25      259  07 13:40:27       1  105560              0  17     7     13   
2   12      212  07 18:05:24       1  101424              0  19     7     18   
3   13      477  07 04:58:08       1   94584              0  13     7      4   
4   12      178  09 09:00:09       1   68413              0   1     9      9   

   minutes  seconds  
0       30       38  
1       40       27  
2        5       24  
3       58        8  
4        0        9

Answer 2

一种解决方案是首先按空格分割，然后转换为datetime个对象，然后直接提取组件。

import pandas as pd

df = pd.DataFrame({'click_time': ['07 09:30:38', '07 13:40:27', '07 18:05:24',
                                  '07 04:58:08', '09 09:00:09', '09 01:22:13',
                                  '09 01:17:58', '07 10:01:53', '08 09:35:17',
                                  '08 12:35:26']})

df[['day', 'time']] = df['click_time'].str.split().apply(pd.Series)
df['datetime'] = pd.to_datetime(df['time'])

df['day'] = df['day'].astype(int)
df['hour'] = df['datetime'].dt.hour
df['minute'] = df['datetime'].dt.minute
df['second'] = df['datetime'].dt.second

df = df.drop(['time', 'datetime'], 1)

<强>结果

    click_time  day  hour  minute  second
0  07 09:30:38    7     9      30      38
1  07 13:40:27    7    13      40      27
2  07 18:05:24    7    18       5      24
3  07 04:58:08    7     4      58       8
4  09 09:00:09    9     9       0       9
5  09 01:22:13    9     1      22      13
6  09 01:17:58    9     1      17      58
7  07 10:01:53    7    10       1      53
8  08 09:35:17    8     9      35      17
9  08 12:35:26    8    12      35      26

如何在庞大的Pandas数据框中分割日，小时，分钟和秒数据？

2 个答案: