Question

假设我有一个带有此数据的别名transactions：

person  store  spent
A       S      3.3
A       S      4.7
B       S      1.2
B       T      3.4

我想知道每个商店有多少不同的人和他们在那里花了多少钱：

store   visitors  revenue
S       2         9.2
T       1         3.4

我希望我能一步到位：

stores = foreach (group transactions by store) generate
  group as store, SUM(transactions.spent) as revenue, 
  COUNT(UNIQUE(transactions.person)) as visitors;

但看起来不像UNIQUE那样。

我是否坚持了两个步骤？

tr1 = foreach (group transactions by (store,person)) generate
  group.store as store, SUM(spent) as revenue;
stores = foreach (group tr1 by store) generate
  group as store, COUNT(tr1) as visitors, SUM(revenue) as revenue;

Answer 1

这里有两种方法

1）使用Distinct内置UDF（不是DISTINCT pig操作符）。对不起，我没有代码示例，我不知道它将如何执行。

2）使用带有DISTINCT运算符的嵌套foreach 像这样的东西：

stores = FOREACH (GROUP transactions BY store) {
    uniqueVisitors = DISTINCT visitors;
    GENERATE
        group AS store,
        COUNT(uniqueVisitors) AS visitors,
        SUM(revenue) AS revenue;
}

关于第二种方法的一个好处是它不应该禁用COMBINER： http://pig.apache.org/docs/r0.11.1/perf.html#When+the+Combiner+is+Used

Answer 2

使用Distinct内置UDF，您只需将UNIQUE替换为org.apache.pig.builtin.Distinct，

stores = foreach (group transactions by store) generate
    group as store, SUM(transactions.spent) as revenue, 
    COUNT(org.apache.pig.builtin.Distinct(transactions.person)) as visitors;

Answer 3

DISTINCT应该做你想要的。

http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html#DISTINCT

计算包中的不同元素

3 个答案: