Question

我为pig script写了wordcount，效果很好。我可以在hdfs的输出目录中看到pig脚本的结果。但是在我的控制台结束时，我看到以下内容：

Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime  MinMapTIme  AvgMapTime  MedianMapTime   MaxReduceTime   MinReduceTime   AvgReduceTime   MedianReducetime    Alias   Feature Outputs
job_local1695568121_0002    1   1   0   0   0   0   0   0   0   0   words_sorted    SAMPLER 
job_local2103470491_0003    1   1   0   0   0   0   0   0   0   0   words_sorted    ORDER_BY    /output/result_pig,
job_local696057848_0001 1   1   0   0   0   0   0   0   0   0   book,words,words_agg,words_grouped  GROUP_BY,COMBINER   

Input(s):
Successfully read 0 records from: "/data/pg5000.txt"

Output(s):
Successfully stored 0 records in: "/output/result_pig"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local696057848_0001 ->  job_local1695568121_0002,
job_local1695568121_0002    ->  job_local2103470491_0003,
job_local2103470491_0003


2014-07-01 14:10:35,241 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

正如您所看到的，工作是成功的。但不是Input(s)和output(s)。他们两个都说successfully read/stored 0 records，计数器值都是0。为什么价值为零。这些不应该是零。

我正在使用hadoop2.2 and pig-0.12

这是脚本：

book = load '/data/pg5000.txt' using PigStorage() as (lines:chararray);
words = foreach book generate FLATTEN(TOKENIZE(lines)) as word;
words_grouped = group words by word;
words_agg = foreach words_grouped generate group as word, COUNT(words);
words_sorted = ORDER words_agg BY $1 DESC;
STORE words_sorted into '/output/result_pig' using PigStorage(':','-schema');

注意：我的数据显示在/data/pg5000.txt中，而不是默认目录/usr/name/data/pg5000.txt

编辑：这是将我的文件打印到控制台的输出

hadoop fs -cat /data/pg5000.txt | head -10
The Project Gutenberg EBook of The Notebooks of Leonardo Da Vinci, Complete
by Leonardo Da Vinci
(#3 in our series by Leonardo Da Vinci)

Copyright laws are changing all over the world. Be sure to check the
copyright laws for your country before downloading or redistributing
this or any other Project Gutenberg eBook.

This header should be the first thing seen when viewing this Project
Gutenberg file.  Please do not remove it.  Do not change or edit the
cat: Unable to write to output stream.

Answer 1

请更正以下行

book = load '/data/pg5000.txt' using PigStorage() as (lines:chararray);

到

book = load '/data/pg5000.txt' using PigStorage(',') as (lines:chararray);

我假设分隔符为逗号，此处使用用于分隔文件中记录的分隔符。这将解决问题

另请注意 -

如果没有提供参数，PigStorage将采用制表符分隔的格式。如果提供了分隔符参数，则它必须是单字节字符;任何文字（例如：＆＃39; a＆＃39;，＃39; |＆＃39;），已知的转义字符（例如：＆＃39; \ t＆＃39;，＆＃39; \ r＆＃39; ）是一个有效的分隔符。

猪拉丁 - 没有显示正确的记录数字

1 个答案: