Question

我在Pig中有两个关系，看起来像这样：

rel_A: {key: chararray, some_string: chararray, some_metric: long}
rel_B: {key: chararray, some_metric2: long}

例如，rel_A看起来像

{('A', 'aaa', 1)
 ('A', 'aab', 2)
 ('B', 'aaa', 3)
 ('B', 'bbb', 1)
 ('C', 'whatever', 5)}

而且rel_B看起来像

{('A', 100)
 ('B', 250)
 ('C', 0)}

我想加入他们，以便我得到：

{('A', 'aaa', 1, 100)
 ('A', 'aab', 2, 100)
 ('B', 'aaa', 3, 250)
 ('B', 'bbb', 1, 250)
 ('C', 'whatever', 5, 0)}

这对我来说在概念上看起来很简单，看起来它只是一个左外连接，但是当我尝试以下操作时遇到了问题：

joined_thing = JOIN rel_A BY key LEFT OUTER, rel_B BY key;

--The error appears here
agged_flat = FOREACH joined_thing GENERATE rel_A::key as key, 
    rel_A::some_string as some_string, 
    rel_A::some_metric as some_metric, 
    rel_B::some_metric2 as some_metric2;

这引发：

Error: org.apache.pig.backend.executionengine.ExecException: ERROR 0:    Scalar has more than one row in the output. 1st : (A,aaa,1), 2nd : (A,aab,2)

我确定我在这里遗漏了一些概念基础，但我一直很努力地尝试做这项工作，非常感谢任何帮助！

Answer 1

解决。以上将有效。

具体错误来自

{('A', 'aaa', 1),
('A', 'aab', 2),
('B', 'aaa', 3),
('B', 'bbb', 1),
('C', 'whatever', 5)}

实际上像嵌套元组一样出现：

{(('A', 'aaa', 1)
('A', 'aab', 2)),
('B', 'aaa', 3),
('B', 'bbb', 1),
('C', 'whatever', 5)}

即使前面提到的脚本刚刚加载过，但其中一个使用了一个自定义库，它实际上有一个我无法检测到的错误。道歉！

Answer 2

按预期工作，可能是负载出错

rel_A = LOAD '/user/data/A_rel.txt' USING PigStorage(',') as (key: chararray, some_string: chararray, some_metric: long);
rel_B = LOAD '/user/data/B_rel.txt' USING PigStorage(',') as  (key: chararray, some_metric2: long);

joined_thing = JOIN rel_A BY key LEFT OUTER, rel_B BY key;


(A,aab,2,A,100)
(A,aaa,1,A,100)
(B,bbb,1,B,250)
(B,aaa,3,B,250)
(C,whatever,5,C,0)

处理一：猪加入的许多关系

2 个答案: