我有一个包含多个列的DataFrame df
。对于这个问题,我只会切出感兴趣的列并将其保存在x
中:
df
是一个巨大的数据框,我从中切割数据:
In [29]: x = df[['date', 'amount', 'price']][:25]
就像x
看起来的信息一样,请看:
In [30]: x
Out[28]:
date amount price
0 2000-11-01 3 57
1 2000-11-01 2 48
2 2000-11-01 1 135
3 2000-11-01 1 24
4 2000-11-01 2 170
5 2000-11-01 1 46
6 2000-11-01 1 28
7 2000-11-01 1 55
8 2000-11-01 1 90
9 2000-11-01 1 20
10 2000-11-01 1 109
11 2000-11-01 1 25
12 2000-11-01 1 129
13 2000-11-01 1 19
14 2000-11-01 1 19
15 2000-11-01 1 168
16 2000-11-01 1 19
17 2000-11-01 1 29
18 2000-11-01 2 48
19 2000-11-01 1 29
20 2000-11-01 1 98
21 2000-11-01 2 58
22 2000-11-01 1 24
23 2000-11-01 2 56
24 2000-11-01 1 86
In [31]: x.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 25 entries, 0 to 24
Data columns (total 3 columns):
date 25 non-null datetime64[ns]
amount 25 non-null int64
price 25 non-null int64
dtypes: datetime64[ns](1), int64(2)
现在我想要一个包含每件商品价格的新列。这是:
price
具有相同的值price/amount
我尝试使用布尔索引:
In [32]: x['price1'] = x['price'] # make a full copy of the column
In [33]: rows = x['amount'] > 1
In [34]: x['price1'][rows] = x['price1'][rows] / x['amount'][rows] # change rows where amount>1
适用于小型数据集x
。
输出如下:
In [54]: x
Out[54]:
date amount price price1
0 2000-11-01 3 57 19
1 2000-11-01 2 48 24
2 2000-11-01 1 135 135
3 2000-11-01 1 24 24
4 2000-11-01 2 170 85
5 2000-11-01 1 46 46
6 2000-11-01 1 28 28
7 2000-11-01 1 55 55
8 2000-11-01 1 90 90
9 2000-11-01 1 20 20
10 2000-11-01 1 109 109
11 2000-11-01 1 25 25
12 2000-11-01 1 129 129
13 2000-11-01 1 19 19
14 2000-11-01 1 19 19
15 2000-11-01 1 168 168
16 2000-11-01 1 19 19
17 2000-11-01 1 29 29
18 2000-11-01 2 48 24
19 2000-11-01 1 29 29
20 2000-11-01 1 98 98
21 2000-11-01 2 58 29
22 2000-11-01 1 24 24
23 2000-11-01 2 56 28
24 2000-11-01 1 86 86
当我用更完整的代码切出更大范围的df
时:
x = df[['date', 'amount', 'price']][:100]
x['price1'] = x['price']
rows = x['amount'] > 1
x['price1'][rows] = x['price'][rows] / x['amount'][rows]
然后我为某些部门获得了NaN:
In [113]: x
Out[113]:
date amount price price1
0 2000-11-01 3 57 19 <<
1 2000-11-01 2 48 24 <<
2 2000-11-01 1 135 135
3 2000-11-01 1 24 24
4 2000-11-01 2 170 NaN
5 2000-11-01 1 46 46
6 2000-11-01 1 28 28
7 2000-11-01 1 55 55
8 2000-11-01 1 90 90
9 2000-11-01 1 20 20
10 2000-11-01 1 109 109
11 2000-11-01 1 25 25
12 2000-11-01 1 129 129
13 2000-11-01 1 19 19
14 2000-11-01 1 19 19
15 2000-11-01 1 168 168
16 2000-11-01 1 19 19
17 2000-11-01 1 29 29
18 2000-11-01 2 48 NaN
19 2000-11-01 1 29 29
20 2000-11-01 1 98 98
21 2000-11-01 2 58 85 <<
22 2000-11-01 1 24 24
23 2000-11-01 2 56 NaN
24 2000-11-01 1 86 86
25 2000-11-01 1 145 145
26 2000-11-01 1 29 29
27 2000-11-01 12 434 NaN
28 2000-11-01 1 46 46
29 2000-11-01 1 52 52
.. ... ... ... ...
70 2000-11-01 1 38 38
71 2000-11-01 1 80 80
72 2000-11-01 1 79 79
73 2000-11-01 2 140 24 <<
74 2000-11-01 1 38 38
75 2000-11-01 1 40 40
76 2000-11-01 3 78 NaN
77 2000-11-01 2 104 NaN
78 2000-11-01 2 130 29 <<
79 2000-11-01 1 96 96
80 2000-11-01 1 42 42
81 2000-11-01 1 109 109
82 2000-11-01 1 89 89
83 2000-11-01 1 26 26
84 2000-11-01 1 49 49
85 2000-11-01 1 135 135
86 2000-11-01 1 38 38
87 2000-11-01 1 29 29
88 2000-11-01 2 46 NaN
89 2000-11-01 1 89 89
90 2000-11-01 1 25 25
91 2000-11-01 2 118 28 <<
92 2000-11-01 1 85 85
93 2000-11-01 1 52 52
94 2000-11-01 1 42 42
95 2000-11-01 2 84 NaN
96 2000-11-01 1 18 18
97 2000-11-01 1 28 28
98 2000-11-01 1 85 85
99 2000-11-01 1 102 102
[100 rows x 4 columns]
奇怪的是,有些部门工作(标有<<
)。
什么想法可能会错误地发生?
感谢
我尝试了一点,当我在分割之前将新的price1
- 列转换为float64
时,它似乎有效。对我来说,这似乎是一个错误。我甚至可以在分割后将其转换回int64
,结果很好。我不知道为什么它适用于小切片(即当我do x = df[...][:25]
时)正确!?
x = df[['date', 'amount', 'price']][:100]
x['price1'] = x['price'].astype(float64)
rows = x['amount'] > 1
x['price1'][rows] = (x['price1'][rows] / x['amount'][rows]).astype(int64)
x
给出:
In [146]: x = df[['date', 'amount', 'price']][:100]
In [147]: x['price1'] = x['price'].astype(float64)
In [148]: rows = x['amount'] > 1
In [149]: x['price1'][rows] = (x['price1'][rows] / x['amount'][rows]).astype(int64)
In [150]: x
Out[150]:
date amount price price1
0 2000-11-01 3 57 19
1 2000-11-01 2 48 24
2 2000-11-01 1 135 135
3 2000-11-01 1 24 24
4 2000-11-01 2 170 85
5 2000-11-01 1 46 46
6 2000-11-01 1 28 28
7 2000-11-01 1 55 55
8 2000-11-01 1 90 90
9 2000-11-01 1 20 20
10 2000-11-01 1 109 109
11 2000-11-01 1 25 25
12 2000-11-01 1 129 129
13 2000-11-01 1 19 19
14 2000-11-01 1 19 19
15 2000-11-01 1 168 168
16 2000-11-01 1 19 19
17 2000-11-01 1 29 29
18 2000-11-01 2 48 24
19 2000-11-01 1 29 29
20 2000-11-01 1 98 98
21 2000-11-01 2 58 29
22 2000-11-01 1 24 24
23 2000-11-01 2 56 28
24 2000-11-01 1 86 86
25 2000-11-01 1 145 145
26 2000-11-01 1 29 29
27 2000-11-01 12 434 36
28 2000-11-01 1 46 46
29 2000-11-01 1 52 52
.. ... ... ... ...
70 2000-11-01 1 38 38
71 2000-11-01 1 80 80
72 2000-11-01 1 79 79
73 2000-11-01 2 140 70
74 2000-11-01 1 38 38
75 2000-11-01 1 40 40
76 2000-11-01 3 78 26
77 2000-11-01 2 104 52
78 2000-11-01 2 130 65
79 2000-11-01 1 96 96
80 2000-11-01 1 42 42
81 2000-11-01 1 109 109
82 2000-11-01 1 89 89
83 2000-11-01 1 26 26
84 2000-11-01 1 49 49
85 2000-11-01 1 135 135
86 2000-11-01 1 38 38
87 2000-11-01 1 29 29
88 2000-11-01 2 46 23
89 2000-11-01 1 89 89
90 2000-11-01 1 25 25
91 2000-11-01 2 118 59
92 2000-11-01 1 85 85
93 2000-11-01 1 52 52
94 2000-11-01 1 42 42
95 2000-11-01 2 84 42
96 2000-11-01 1 18 18
97 2000-11-01 1 28 28
98 2000-11-01 1 85 85
99 2000-11-01 1 102 102
[100 rows x 4 columns]
答案 0 :(得分:2)
你正在进行链式作业,你不应该这样做,因为它有时不起作用,你正在观察的是:http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
此外,如果可能的话,你应该总是使用.loc
,如果我们比较使用掩码和没有掩码之间的性能,我们可以看到,对于基于样本数据的25000行数据帧,没有掩码时它会更快:
In [17]:
%%timeit
x = df[['date', 'amount', 'price']][:100]
x['price1'] = x['price']
rows = x['amount'] > 1
x.loc[rows,'price1']= x['price'] / x['amount']
100 loops, best of 3: 2.54 ms per loop
In [19]:
%timeit x.loc[rows,'price1']= x['price'] / x['amount']
1000 loops, best of 3: 950 µs per loop
您的原始代码:
In [23]:
%%timeit
x = df[['date', 'amount', 'price']][:100]
x['price1'] = x['price'].astype(float64)
rows = x['amount'] > 1
x['price1'][rows] = (x['price1'][rows] / x['amount'][rows]).astype(int64)
100 loops, best of 3: 2.48 ms per loop
所以你看到划分整个数据帧比选择前100行更快,然后屏蔽再划分