Question

我怎样才能找到一个领域的连续性和开始位置输入就像

A-1
B-2
B-3
B-4
C-5
C-6

我想要的输出是

A,1,1
B,3,2
C,2,5

谢谢。

Answer 1

对continous_counts的值进行分组和计数。即

A,1
B,3
C,2

获取每个值的第一行。即

A,1
B,2
C,5

加入以上两个关系并获得所需的输出。

A = LOAD 'data.txt' USING PigStorage('-') AS (value:chararray;index:int);
B = GROUP A BY value;
C = FOREACH B GENERATE group as value,COUNT(A.value) as continuous_counts;
D = FOREACH B {
    ordered = ORDER B BY index;
    first = LIMIT ordered 1;
    GENERATE first.value,first.index;
    }
E = JOIN C BY value,D BY value;
F = FOREACH E GENERATE C::value,C::continuous_counts,D::index;
DUMP F;

Answer 2

假设您没有关于value的不连续数据，则可以通过首先对value进行分组并使用COUNT和MIN来获得所需的结果分别为continuous_counts和start_index。

A = LOAD 'data' USING PigStorage('-') AS (value:chararray;index:int);

B = FOREACH (GROUP A BY value) GENERATE
    group as value,
    COUNT(A) as continuous_counts,
    MIN(A.value) as start_index;

STORE B INTO 'output' USING PigStorage(',');

如果您的数据确实有可能出现不连续的数据，则该解决方案在本地猪中不再是琐碎的事，您可能需要为此编写UDF。

查找Pig中元素的连续性

2 个答案: