Question

架构：

TYP|ID|RECORD|SEX|EFF_DT|CANC_DT

DMF|1234567|98765432|M|2011-08-30|9999-12-31
DMF|1234567|98765432|M|2011-04-30|9999-12-31
DMF|1234567|98765432|M|2011-04-30|9999-12-31

假设我有多条这样的记录。我只想显示具有最小eff_dt和最大取消日期的记录。

我只想显示这1条记录

DMF|1234567|98765432|M|2011-04-30|9999-12-31

谢谢

Answer 1

获取min eff_dt和max canc_dt并使用它来过滤关系。假设你有一个关系A

B = GROUP A ALL;
X = FOREACH B GENERATE MIN(A.EFF_DT);
Y = FOREACH B GENERATE MAX(A.CANC_DT);

C = FILTER A BY ((EFF_DT == X.$0) AND (CANC_DT == Y.$0));
D = DISTINCT C;
DUMP D;

Answer 2

我们假设你有这些数据（这里有样本）：

DMF|1234567|98765432|M|2011-08-30|9999-12-31
DMF|1234567|98765432|M|2011-04-30|9999-12-31
DMF|1234567|98765432|M|2011-04-30|9999-12-31
DMX|1234567|98765432|M|2011-12-30|9999-12-31
DMX|1234567|98765432|M|2011-04-30|9999-12-31
DMX|1234567|98765432|M|2011-04-01|9999-12-31

执行以下步骤：

-- 1. Read data, if you have not 
 A = load 'data.txt' using PigStorage('|') as (typ: chararray, id:chararray, record:chararray, sex:chararray, eff_dt:datetime, canc_dt:datetime);

-- 2. Group data by the attribute you like to, in this case it is TYP
grouped = group A by typ;

-- 3. Now, generate MIN/MAX for each group. Also, only keep relevant fields
min_max = foreach grouped generate group, MIN(A.eff_dt) as min_eff_dt, MAX(A.canc_dt) as max_canc_dt;

-- 
dump min_max;
(DMF,2011-04-30T00:00:00.000Z,9999-12-31T00:00:00.000Z)
(DMX,2011-04-01T00:00:00.000Z,9999-12-31T00:00:00.000Z)

如果需要，请将日期时间更改为charrary。

注意：有不同的方法，我正在展示的内容，除了加载步骤，它分两步产生所需的结果：GROUP和FOREACH。

从PIG中的数据中获取MIN EFF_DT和MAX_CANC_dt

2 个答案: