Question

我来自MATLAB背景，我可以简单地做到这一点

age_sum_error = sum(age > prediction - 4 & age < prediction + 4);

这将计算age为真的prediction (+4/-4)值的数量，我想在spark数据框中执行类似的操作。

说这是我的火花数据框

+--------------------------+
|age | gender | prediction |
+----+--------+------------+
|35  |  M     | 30         |
|40  |  F     | 42         |
|45  |  F     | 38         |
|26  |  F     | 29         |
+----+--------+------------+

我希望我的结果看起来像这样

+------+----------+
|false | positive |
+------+----------+
|2     | 2        |
+------+----------+

Answer 1

首先计算条件，然后通过汇总 1 s 和 0 s ：

df.selectExpr(
    'cast(abs(age - prediction) < 4 as int) as condition'
).selectExpr(
    'sum(condition) as positive', 
    'sum(1-condition) as negative'
).show()
+--------+--------+
|positive|negative|
+--------+--------+
|       2|       2|
+--------+--------+

Answer 2

它的代码比matlab多得多，但是这就是我的方法。

import numpy as np

ages = [35, 40, 45, 26]
pred = [30, 42, 38, 29]
tolerance = 4

# get boolean array of people older and younger than limits
is_older = np.greater(ages, pred-tolerance) # a boolean array
is_younger =  np.less(ages, pred+tolerance) # a boolean array

# convert these boolean arrays to ints then multiply. True = 1, False = 0. 
in_range = is_older.astype(int)*is_younger.astype(int) # 0's cancel 1's

# add upp the indixes that are still 1
senior_count = np.sum(in_range)

希望这会有所帮助。

计算火花数据框中的真假条件数

2 个答案: