Python / Pandas:元素分裂导致NaN的错误?

时间:2014-06-11 09:02:13

标签: python debugging pandas dataframe division

背景&数据看起来像什么:

我有一个包含多个列的DataFrame df。对于这个问题,我只会切出感兴趣的列并将其保存在x中:

df是一个巨大的数据框,我从中切割数据:

In [29]: x = df[['date', 'amount', 'price']][:25]

就像x看起来的信息一样,请看:

In [30]: x
Out[28]:
     date      amount  price
0  2000-11-01       3     57
1  2000-11-01       2     48
2  2000-11-01       1    135
3  2000-11-01       1     24
4  2000-11-01       2    170
5  2000-11-01       1     46
6  2000-11-01       1     28
7  2000-11-01       1     55
8  2000-11-01       1     90
9  2000-11-01       1     20
10 2000-11-01       1    109
11 2000-11-01       1     25
12 2000-11-01       1    129
13 2000-11-01       1     19
14 2000-11-01       1     19
15 2000-11-01       1    168
16 2000-11-01       1     19
17 2000-11-01       1     29
18 2000-11-01       2     48
19 2000-11-01       1     29
20 2000-11-01       1     98
21 2000-11-01       2     58
22 2000-11-01       1     24
23 2000-11-01       2     56
24 2000-11-01       1     86

In [31]: x.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 25 entries, 0 to 24
Data columns (total 3 columns):
date      25 non-null datetime64[ns]
amount    25 non-null int64
price     25 non-null int64
dtypes: datetime64[ns](1), int64(2)

我想要什么:

现在我想要一个包含每件商品价格的新列。这是:

  • 当金额== 1 - &gt;它与该行中的price具有相同的值
  • 当金额&gt; 1 - &gt;它是该行的price/amount

我尝试使用布尔索引:

In [32]: x['price1'] = x['price'] # make a full copy of the column

In [33]: rows = x['amount'] > 1

In [34]: x['price1'][rows] = x['price1'][rows] / x['amount'][rows] # change rows where amount>1

适用于小型数据集x。 输出如下:

In [54]: x
Out[54]:
         date  amount  price  price1
0  2000-11-01       3     57      19
1  2000-11-01       2     48      24
2  2000-11-01       1    135     135
3  2000-11-01       1     24      24
4  2000-11-01       2    170      85
5  2000-11-01       1     46      46
6  2000-11-01       1     28      28
7  2000-11-01       1     55      55
8  2000-11-01       1     90      90
9  2000-11-01       1     20      20
10 2000-11-01       1    109     109
11 2000-11-01       1     25      25
12 2000-11-01       1    129     129
13 2000-11-01       1     19      19
14 2000-11-01       1     19      19
15 2000-11-01       1    168     168
16 2000-11-01       1     19      19
17 2000-11-01       1     29      29
18 2000-11-01       2     48      24
19 2000-11-01       1     29      29
20 2000-11-01       1     98      98
21 2000-11-01       2     58      29
22 2000-11-01       1     24      24
23 2000-11-01       2     56      28
24 2000-11-01       1     86      86

问题:

当我用更完整的代码切出更大范围的df时:

x = df[['date', 'amount', 'price']][:100]
x['price1'] = x['price']
rows = x['amount'] > 1
x['price1'][rows] = x['price'][rows] / x['amount'][rows]

然后我为某些部门获得了NaN:

In [113]: x                         
Out[113]:                           
         date  amount  price  price1
0  2000-11-01       3     57      19 <<
1  2000-11-01       2     48      24 <<
2  2000-11-01       1    135     135
3  2000-11-01       1     24      24
4  2000-11-01       2    170     NaN 
5  2000-11-01       1     46      46
6  2000-11-01       1     28      28
7  2000-11-01       1     55      55
8  2000-11-01       1     90      90
9  2000-11-01       1     20      20
10 2000-11-01       1    109     109
11 2000-11-01       1     25      25
12 2000-11-01       1    129     129
13 2000-11-01       1     19      19
14 2000-11-01       1     19      19
15 2000-11-01       1    168     168
16 2000-11-01       1     19      19
17 2000-11-01       1     29      29
18 2000-11-01       2     48     NaN
19 2000-11-01       1     29      29
20 2000-11-01       1     98      98
21 2000-11-01       2     58      85 <<
22 2000-11-01       1     24      24
23 2000-11-01       2     56     NaN
24 2000-11-01       1     86      86
25 2000-11-01       1    145     145
26 2000-11-01       1     29      29
27 2000-11-01      12    434     NaN
28 2000-11-01       1     46      46
29 2000-11-01       1     52      52
..        ...     ...    ...     ...
70 2000-11-01       1     38      38
71 2000-11-01       1     80      80
72 2000-11-01       1     79      79
73 2000-11-01       2    140      24 <<
74 2000-11-01       1     38      38
75 2000-11-01       1     40      40
76 2000-11-01       3     78     NaN
77 2000-11-01       2    104     NaN
78 2000-11-01       2    130      29 <<
79 2000-11-01       1     96      96
80 2000-11-01       1     42      42
81 2000-11-01       1    109     109
82 2000-11-01       1     89      89
83 2000-11-01       1     26      26
84 2000-11-01       1     49      49
85 2000-11-01       1    135     135
86 2000-11-01       1     38      38
87 2000-11-01       1     29      29
88 2000-11-01       2     46     NaN
89 2000-11-01       1     89      89
90 2000-11-01       1     25      25
91 2000-11-01       2    118      28 <<
92 2000-11-01       1     85      85
93 2000-11-01       1     52      52
94 2000-11-01       1     42      42
95 2000-11-01       2     84     NaN
96 2000-11-01       1     18      18
97 2000-11-01       1     28      28
98 2000-11-01       1     85      85
99 2000-11-01       1    102     102                                        
[100 rows x 4 columns]              

奇怪的是,有些部门工作(标有<<)。 什么想法可能会错误地发生? 感谢

更加深刻的洞察力&#34; BUG&#34;?

我尝试了一点,当我在分割之前将新的price1 - 列转换为float64时,它似乎有效。对我来说,这似乎是一个错误。我甚至可以在分割后将其转换回int64,结果很好。我不知道为什么它适用于小切片(即当我do x = df[...][:25]时)正确!?

x = df[['date', 'amount', 'price']][:100]
x['price1'] = x['price'].astype(float64)
rows = x['amount'] > 1
x['price1'][rows] = (x['price1'][rows] / x['amount'][rows]).astype(int64)
x

给出:

In [146]: x = df[['date', 'amount', 'price']][:100]

In [147]: x['price1'] = x['price'].astype(float64)

In [148]: rows = x['amount'] > 1

In [149]: x['price1'][rows] = (x['price1'][rows] / x['amount'][rows]).astype(int64)

In [150]: x
Out[150]:
         date  amount  price  price1
0  2000-11-01       3     57      19
1  2000-11-01       2     48      24
2  2000-11-01       1    135     135
3  2000-11-01       1     24      24
4  2000-11-01       2    170      85
5  2000-11-01       1     46      46
6  2000-11-01       1     28      28
7  2000-11-01       1     55      55
8  2000-11-01       1     90      90
9  2000-11-01       1     20      20
10 2000-11-01       1    109     109
11 2000-11-01       1     25      25
12 2000-11-01       1    129     129
13 2000-11-01       1     19      19
14 2000-11-01       1     19      19
15 2000-11-01       1    168     168
16 2000-11-01       1     19      19
17 2000-11-01       1     29      29
18 2000-11-01       2     48      24
19 2000-11-01       1     29      29
20 2000-11-01       1     98      98
21 2000-11-01       2     58      29
22 2000-11-01       1     24      24
23 2000-11-01       2     56      28
24 2000-11-01       1     86      86
25 2000-11-01       1    145     145
26 2000-11-01       1     29      29
27 2000-11-01      12    434      36
28 2000-11-01       1     46      46
29 2000-11-01       1     52      52
..        ...     ...    ...     ...
70 2000-11-01       1     38      38
71 2000-11-01       1     80      80
72 2000-11-01       1     79      79
73 2000-11-01       2    140      70
74 2000-11-01       1     38      38
75 2000-11-01       1     40      40
76 2000-11-01       3     78      26
77 2000-11-01       2    104      52
78 2000-11-01       2    130      65
79 2000-11-01       1     96      96
80 2000-11-01       1     42      42
81 2000-11-01       1    109     109
82 2000-11-01       1     89      89
83 2000-11-01       1     26      26
84 2000-11-01       1     49      49
85 2000-11-01       1    135     135
86 2000-11-01       1     38      38
87 2000-11-01       1     29      29
88 2000-11-01       2     46      23
89 2000-11-01       1     89      89
90 2000-11-01       1     25      25
91 2000-11-01       2    118      59
92 2000-11-01       1     85      85
93 2000-11-01       1     52      52
94 2000-11-01       1     42      42
95 2000-11-01       2     84      42
96 2000-11-01       1     18      18
97 2000-11-01       1     28      28
98 2000-11-01       1     85      85
99 2000-11-01       1    102     102

[100 rows x 4 columns]

1 个答案:

答案 0 :(得分:2)

你正在进行链式作业,你不应该这样做,因为它有时不起作用,你正在观察的是:http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

此外,如果可能的话,你应该总是使用.loc,如果我们比较使用掩码和没有掩码之间的性能,我们可以看到,对于基于样本数据的25000行数据帧,没有掩码时它会更快:

In [17]:

%%timeit
x = df[['date', 'amount', 'price']][:100]
x['price1'] = x['price']
rows = x['amount'] > 1
x.loc[rows,'price1']= x['price'] / x['amount']
100 loops, best of 3: 2.54 ms per loop

In [19]:   
%timeit x.loc[rows,'price1']= x['price'] / x['amount']
1000 loops, best of 3: 950 µs per loop

您的原始代码:

In [23]:

%%timeit
x = df[['date', 'amount', 'price']][:100]
x['price1'] = x['price'].astype(float64)
rows = x['amount'] > 1
x['price1'][rows] = (x['price1'][rows] / x['amount'][rows]).astype(int64)
100 loops, best of 3: 2.48 ms per loop

所以你看到划分整个数据帧比选择前100行更快,然后屏蔽再划分