访问日志Pig Job。数不胜数

Question

我有HTTP访问日志的TB，我正在尝试分析以供历史参考。我正在尝试获取每日访客（IP）的数量和每日点击次数。

如果是在MySQL中我会做这样的事情：

SELECT COUNT(1) from access_logs union
SELECT COUNT(DISTINCT(ip)) from access_logs

任何方式与一份工作做类似的事情，所以我不必第二次做地图工作？

我知道这不起作用，但这是我正在寻找的功能。

X = (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
Y = COUNT(X);
Z = COUNT(DISTINCT(X.IP);
OUT = UNION Y,Z;
STORE OUT into ...

Answer 1

如果要计算输入中的所有记录，则需要使用GROUP ALL，这将创建一个包。同样出于性能原因，它使用累加器DISTINCT函数org.apache.pig.builtin.Distinct

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
GRP = group IPs all;
OUT = foreach GRP generate COUNT(IPs) as all_cnt, COUNT(org.apache.pig.builtin.Distinct(IPs.ip)) as distinct_cnt;

如果你有太多的IP并且你得到了与内存相关的异常，那么你可以做这样的事情：

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
Dist_IPs = distinct IPs;
GRP_DIST = group Dist_IPs all;
DIST = foreach GRP_DIST generate COUNT(GRP_DIST) as cnt, 'dist' as category;

GRP_ALL = group IPs all;
ALL = foreach GRP_ALL generate COUNT(GRP_ALL)as cnt, 'all' as category;

OUT = union DIST, ALL;