Question

假设我有一组目标词：

a b c d

和输入文件：

a d f s g e
12399
c a d i f
a 2

然后我应该回来：

a 3
b 0
c 1
d 2

我怎么能在猪身上做到这一点？谢谢！

Answer 1

首先从每行中删除重复的单词，然后运行单词计数猪步骤：

REGISTER 'udf-1.0-SNAPSHOT.jar'
define tuple_set com.ts.pig.UniqueRecords();
data = load '<file>' using PigStorage();

从每行删除重复的字词

unique= foreach data generate tuple_set($0) as line;
words= foreach unique generate flatten(TOKENIZE(line,' ')) as word;
grouped = group words BY word;
count= foreach grouped GENERATE group, COUNT(words);
dump count;

Pig UDF示例代码：

/**
 * This udf removes duplicate words from line
 */
public class UniqueRecords extends EvalFunc<String> {
    @Override
    public String exec(Tuple tuple) throws IOException {
        if (tuple == null || tuple.size() == 0)
            return null;
        String[] splits=tuple.get(0).toString().split(" ");
        Set<String> elements = new HashSet<String>(Arrays.asList(splits));
        StringBuilder sb = new StringBuilder();
        for(String element:elements ){
            sb.append(element+" ");
        }
        return sb.toString();
    }
}

在pig中，如何计算包含特定字符串的行数？

1 个答案: