Question

我在HDFS文件夹中有几个CSV文件，我加载到以下关系：

source = LOAD'$ data'使用PigStorage（'，'）; - $ data作为参数传递给pig命令。

当我转储它时，源关系的结构如下:(注意数据是文本限定的，但我将使用REPLACE函数处理它）

("HEADER","20110118","20101218","20110118","T00002")
("0000000000000000035412","20110107","2699","D","20110107","2315.","","","","","","C")
("0000000000000000035412","20110107","2699","D","20110107","246..","162","74","","","","B")

<.... more records ....>

("HEADER","20110224","20110109","20110224","T00002")
("0000000000000000035412","20110121","2028","D","20110121","a6c3.","","","","","R","P")
("0000000000000000035412","20110217","2619","D","20110217","a6c3.","","","","","R","P")

<.... more records ....>

因此每个文件都有一个标题，提供有关其后的数据集的一些信息，例如数据提供者及其涵盖的日期范围。

现在，我如何转换上述结构并创建如下所示的新关系？：

{
(HEADER,20110118,20101218,20110118,T00002),{(0000000000000000035412,20110107,2699,D,20110107,2315.,,,,,,C),(0000000000000000035412,20110107,2699,D,20110107,246..,162,74,,,,B),..more tuples..},
(HEADER,20110224,20110109,20110224,T00002),{(0000000000000000035412,20110121,2028,D,20110121,a6c3.,,,,,R,P),(0000000000000000035412,20110217,2619,D,20110217,a6c3.,,,,,R,P),..more tuples..},..more tuples..
}

每个标头元组后面跟着一包属于该标题的记录元组？遗憾的是，标题行和详细信息行之间没有公共关键字段，所以我认为不能使用任何JOIN操作。？

我对Pig和Hadoop很陌生，这是我参与的第一个概念项目之一。

希望我的问题很明确，并期待在此提供一些指导。

Answer 1

这应该让你开始。
代码：

Source = LOAD '$data' USING PigStorage(',','-tagFile');
A = SPLIT Source INTO FileHeaders IF $1 == 'HEADER', FileData OTHERWISE;
B = GROUP FileData BY $0;
C = GROUP FileHeaders BY $0;
D = JOIN B BY Group, C BY Group;
...

如何使用apache pig将标题行连接到多个文件中的详细行

1 个答案: