数学内袋

Question

我有一袋元组，每个包中的一个字段需要归一化为零。我将那个字段的MIN放在包上，然后从每个元组中减去该最小值。

  

这可以不展平吗？

实际情况稍微复杂一点，因为我只希望min超过元组子集满足一定条件。

以下是一些不起作用的示例代码：

data = LOAD 'data.csv' USING PigStorage(',')
    AS (x:int, y:int, z:int);

data_grouped = GROUP data BY x;

data_normal = FOREACH data_grouped {
    good_data = FILTER data BY y == 0;
    smallest_good_z = MIN(good_data.z);
    GENERATE data.(x, y, z-smallest_good_z);
}

DESCRIBE data_normal;

rmf data_normal
STORE data_normal INTO 'data_normal' USING PigStorage(',');

和示例data.csv：

0,0,1
0,0,2
0,0,3
0,1,0
0,2,-1
1,2,3
1,3,4
1,4,5
1,0,5

请告诉我，我不必分组，MIN，展平，减去和重新组合！这是我现在使用的方法，我想摆脱它：

data = LOAD 'data.csv' USING PigStorage(',') AS
    (x:int, y:int, z:int);

data_grouped = GROUP data BY x;

data_n0 = FOREACH data_grouped {
    good_data = FILTER data BY y == 0;
    smallest_good_z = MIN(good_data.z);
    GENERATE FLATTEN(data.(x, y, z)), smallest_good_z AS smz:int;
}

data_n1 = FOREACH data_n0 GENERATE x,y,z-smz;

data_normal = GROUP data_n1 BY x;

Answer 1

不幸的是，你只能用UDF来做。这是一个例子：

import java.io.IOException;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.DataBag;
import org.apache.pig.data.Tuple;

public class MinusToAllInBag extends EvalFunc<Tuple> {

    @Override
    public Tuple exec(Tuple input) throws IOException {
        if (input == null || input.size() != 3) {
            System.err.println("Inputs are ({inputBag}, position, toSubtract)");
            return null;
        }
        try {
            Object o = input.get(0);
            if (!(o instanceof DataBag)) {
                throw new RuntimeException("parameter 1 must be a databag");
            }
            DataBag inputBag = (DataBag)o;
            Integer pos = (Integer) input.get(1);
            Float toSubtract = (Float) input.get(2);
            for (Tuple row : inputBag) {
                Float value = (Float)row.get(pos);
                if (value != null) {
                    value -= toSubtract;
                    row.set(pos, value);
                }
            }
            return input;
        } catch (Exception e) {
            System.err.println("Failed to process input; error - " + e.getMessage());
            return null;
        }
    }
}

和猪脚本：

REGISTER libs.jar;

data = LOAD 'data.csv' USING PigStorage(',') AS
    (x:int, y:int, z:float);

data_grouped = GROUP data BY x;

data_n0 = FOREACH data_grouped {
    good_data = FILTER data BY y == 0;
    smallest_good_z = MIN(good_data.z);
    GENERATE group, MinusToAllInBag(data, 2, (float)smallest_good_z);
}

dump data_n0;