Question

我的数据如下：

            Close   a   b   c   d   e   Time    
2015-12-03  2051.25 5   4   3   1   1   05:00:00    
2015-12-04  2088.25 5   4   3   1   NaN 06:00:00
2015-12-07  2081.50 5   4   3   NaN NaN 07:00:00
2015-12-08  2058.25 5   4   NaN NaN NaN 08:00:00
2015-12-09  2042.25 5   NaN NaN NaN NaN 09:00:00

我需要“水平”计算不是NaN的列['a']到['e']中的值。结果就是：

df['Count'] = .....
df

            Close   a   b   c   d   e   Time     Count
2015-12-03  2051.25 5   4   3   1   1   05:00:00 5  
2015-12-04  2088.25 5   4   3   1   NaN 06:00:00 4
2015-12-07  2081.50 5   4   3   NaN NaN 07:00:00 3
2015-12-08  2058.25 5   4   NaN NaN NaN 08:00:00 2
2015-12-09  2042.25 5   NaN NaN NaN NaN 09:00:00 1

由于

Answer 1

您可以从df中进行选择，并通过count致电axis=1：

In [24]:
df['count'] = df[list('abcde')].count(axis=1)
df

Out[24]:
              Close  a   b   c   d   e      Time  count
2015-12-03  2051.25  5   4   3   1   1  05:00:00      5
2015-12-04  2088.25  5   4   3   1 NaN  06:00:00      4
2015-12-07  2081.50  5   4   3 NaN NaN  07:00:00      3
2015-12-08  2058.25  5   4 NaN NaN NaN  08:00:00      2
2015-12-09  2042.25  5 NaN NaN NaN NaN  09:00:00      1

<强>的时间设置

In [25]:
%timeit df[['a', 'b', 'c', 'd', 'e']].apply(lambda x: sum(x.notnull()), axis=1)
%timeit df.drop(['Close', 'Time'], axis=1).count(axis=1)
%timeit df[list('abcde')].count(axis=1)

100 loops, best of 3: 3.28 ms per loop
100 loops, best of 3: 2.76 ms per loop
100 loops, best of 3: 2.98 ms per loop

apply是最慢的，这并不令人意外，drop版本稍微快一些，但在语义上我更喜欢传递感兴趣的列表并调用count以获取可读性

嗯，我现在一直在改变时间：

In [27]:
%timeit df[['a', 'b', 'c', 'd', 'e']].apply(lambda x: sum(x.notnull()), axis=1)
%timeit df.drop(['Close', 'Time'], axis=1).count(axis=1)
%timeit df[list('abcde')].count(axis=1)
%timeit df[['a', 'b', 'c', 'd', 'e']].count(axis=1)

100 loops, best of 3: 3.33 ms per loop
100 loops, best of 3: 2.7 ms per loop
100 loops, best of 3: 2.7 ms per loop
100 loops, best of 3: 2.57 ms per loop

更多时间

In [160]:
%timeit df[['a', 'b', 'c', 'd', 'e']].apply(lambda x: sum(x.notnull()), axis=1)
%timeit df.drop(['Close', 'Time'], axis=1).count(axis=1)
%timeit df[list('abcde')].count(axis=1)
%timeit df[['a', 'b', 'c', 'd', 'e']].count(axis=1)
%timeit df[list('abcde')].notnull().sum(axis=1) 

1000 loops, best of 3: 1.4 ms per loop
1000 loops, best of 3: 1.14 ms per loop
1000 loops, best of 3: 1.11 ms per loop
1000 loops, best of 3: 1.11 ms per loop
1000 loops, best of 3: 1.05 ms per loop

似乎测试notnull和求和（如notnull将生成布尔掩码）在此数据集上更快

在50k行上df，最后一种方法稍微快一些：

In [172]:
%timeit df[['a', 'b', 'c', 'd', 'e']].apply(lambda x: sum(x.notnull()), axis=1)
%timeit df.drop(['Close', 'Time'], axis=1).count(axis=1)
%timeit df[list('abcde')].count(axis=1)
%timeit df[['a', 'b', 'c', 'd', 'e']].count(axis=1)
%timeit df[list('abcde')].notnull().sum(axis=1) 

1 loops, best of 3: 5.83 s per loop
100 loops, best of 3: 6.15 ms per loop
100 loops, best of 3: 6.49 ms per loop
100 loops, best of 3: 6.04 ms per loop

Answer 2

包括所需的columns列表，或者只是删除您不希望从计数中排除的两个columns - axis=1 (see docs)：

df['Count'] = df.drop(['Close', 'Time'], axis=1).count(axis=1)


     Close  a  b   c   d   e      Time  Count
0  2051.25  5  4   3   1   1  05:00:00      5
1  2088.25  5  4   3   1 NaN  06:00:00      4
2  2081.50  5  4   3 NaN NaN  07:00:00      3
3  2058.25  5  4   3 NaN NaN  08:00:00      3
4  2042.25  5  4 NaN NaN NaN  09:00:00      2

Answer 3

df['Count'] = df[['a', 'b', 'c', 'd', 'e']].apply(lambda x: sum(x.notnull()), axis=1)

In [1254]: df
Out[1254]: 
              Close  a   b   c   d   e      Time  Count
2015-12-03  2051.25  5   4   3   1   1  05:00:00      5
2015-12-04  2088.25  5   4   3   1 NaN  06:00:00      4
2015-12-07  2081.50  5   4   3 NaN NaN  07:00:00      3
2015-12-08  2058.25  5   4 NaN NaN NaN  08:00:00      2
2015-12-09  2042.25  5 NaN NaN NaN NaN  09:00:00      1

如何在pandas数据帧中的列中计算非NaN值？

3 个答案: