Question

将这些视为我的输入文件，

Database

我将阅读＆＃34;输入1＆＃34;文件使用我的PIG脚本，我希望得到如下结果，基于＆＃34;输入2＆＃34;文件。

 Input 1: (File 1)
 12,23,14,15,9
 1,2,3,4,5
 34,17,8
 .
 .

 Input 2: (File 2)
 12 Twelve
 23 TwentyThree
 34 ThirtyFour
 .
 .

没有UDF可以实现这一目标吗？请让我知道你的建议。

提前致谢！

Answer 1

这违反了“没有UDF”的标准。但UDF是内置的，所以我怀疑它就足够了。

<强>查询：

data1 = LOAD 'file1' AS (val:chararray);
data2 = LOAD 'file2' AS (num:chararray, desc:chararray);

A = RANK data1;  /* creates row number*/
B = FOREACH A GENERATE rank_data1, FLATTEN(TOKENIZE(val, ',')) AS num;
C = RANK B;  /* used to keep tuple elements sorted in bag*/
D = JOIN C BY num, data2 BY num;
E = FOREACH D GENERATE C::rank_data1 AS rank_1:long
                     , C::rank_B AS rank_2:long
                     , data2::desc AS description;

grpd = GROUP E BY rank_1;
F = FOREACH grpd {
      sorted = ORDER E BY rank_2;
      GENERATE sorted;
    };

X = FOREACH F GENERATE FLATTEN(BagToTuple(sorted.description));
DUMP X;

<强>输出：

(Twelve,TwentyThree,Fourteen,Fifteen,Nine)
(One,Two,Three,Four,Five)
(ThirtyFour,Seventeen,Eight)

Answer 2

这是一个Hive解决方案：

--Load the data into Hive
CREATE TABLE file1 (
  line array<string>
)
ROW FORMAT DELIMITED 
COLLECTION ITEMS TERMINATED BY ',';
LOAD DATA INPATH '/tmp/test2/file1' OVERWRITE INTO TABLE  file1;

CREATE TABLE file2 (
  name string,
  value string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ';
LOAD DATA INPATH '/tmp/test2/file2' OVERWRITE INTO TABLE  file2;

--explode the rows from the first table and create a newid to use for correlation
CREATE TABLE file1_exploded 
AS
WITH tmp 
AS
(SELECT RAND() newid, line from file1)
SELECT newid, item FROM tmp 
LATERAL VIEW EXPLODE (line) a AS item;

--apply substitions using the second table, then join lines back together
SELECT CONCAT_WS(',', COLLECT_LIST(value))
FROM 
file1_exploded
JOIN file2 ON item = name
GROUP BY newid;

如何在不使用UDF的情况下获得PIG中的匹配值？

2 个答案: