Question

我正在对Google BigQuery中的publicdata：samples.github_timeline数据集进行漏斗分析。我想按时间顺序提取所有做过一系列三个事件的独特用户。

事件及其顺序：

WatchEvent
PushEvent
CreateEvent

这是查询：

    select user from (
    SELECT user1 as user,
        ts1 as eventDate1,
        ts2 as eventDate2,
        IF(ts2 < ts3, ts3, NULL) as eventDate3
     FROM
        (SELECT user1,
        ts1,
            ts2,
            ts3
            FROM (SELECT user1,
    ts1,
    IF(ts1 < ts2, ts2, NULL) as ts2
    FROM
    (SELECT user1,
    ts1,
    ts2
    FROM (SELECT repository_owner as user1,
    created_at as ts1
    FROM [publicdata:samples.github_timeline]
    WHERE type = "WatchEvent") as step1
    LEFT JOIN EACH (SELECT repository_owner as user2,
    created_at as ts2
    FROM [publicdata:samples.github_timeline]
    WHERE type = "PushEvent") as step2
    ON user1 = user2 where ts1 is not NULL)

    ) as steps1_2
           LEFT JOIN (SELECT repository_owner as user3,
            created_at as ts3
     FROM [publicdata:samples.github_timeline]
     WHERE type = "CreateEvent") as step3
           ON user1 = user3 
           where ts2 is not NULL
           )
    )
    where eventDate3 is not null
    group by user
    limit 100

最后没有GROUP BY用户，它很快（10秒）。但是当我添加它时，需要花费很多时间（超过20分钟）。

查询有什么问题？您可以在此处测试查询：https://bigquery.cloud.google.com/

Answer 1

你有一个加入爆炸;也就是说，如果用户A有20个WatchEvents，20个PushEvents和20个CreateEvents，那么您的查询可以从这60个中生成8000行。这是因为当JOIN两侧有多个匹配键时，它会生成笛卡儿积分双方。您可以通过仅采用最小匹配时间来解决此问题，因此您只需查看最短的WatchEvent时间，以便用户查找后续的PushEvent时间，然后查看比WatchEvent时间晚的最小pushEvent时间来查找匹配CreateEvent时间。

这是一个大约20秒内运行的查询：

SELECT user 
FROM (
  SELECT step2_2.user1 as user,
    MIN(step2_2.ts1) as eventDate1,
    MIN(step2_2.ts2) as eventDate2,
    MIN(step3.ts3) as eventDate3
  FROM (
   SELECT user1, MIN(ts1) as ts1, MIN(ts2) as ts2
   FROM (
    SELECT repository_owner as user1,
    MIN(created_at) as ts1
    FROM [publicdata:samples.github_timeline]
    WHERE type = "WatchEvent"
    GROUP EACH BY user1) as step1
   JOIN EACH (
    SELECT repository_owner as user2,
      created_at as ts2
    FROM [publicdata:samples.github_timeline]
    WHERE type = "PushEvent") as step2
   ON user1 = user2
   WHERE ts1 < ts2
   GROUP EACH BY user1
  ) as step2_2
  JOIN EACH (
    SELECT repository_owner as user3,
      created_at as ts3
    FROM [publicdata:samples.github_timeline]
    WHERE type = "CreateEvent") as step3
  ON user1 = user3 
  WHERE step2_2.ts2 < step3.ts3
  GROUP EACH BY user
  ) 
GROUP BY user  
LIMIT 100

Answer 2

如果您的数据集不是太大，您可以使用lead（）窗口函数来查找序列并完全避免连接。

Select repository_owner 
FROM
(
Select repository_owner,type as Event0, 
LEAD(x,1) OVER(Partition by repository_owner order by ts) as Event1, 
LEAD(x,2) OVER(Partition by repository_owner order by ts) as Event2, 

FROM
(
SELECT repository_owner as user,created_at as ts,type as x
from [publicdata:samples.github_timeline]
where type in ("WatchEvent","PushEvent","CreateEvent")
))
where Event0="WatchEvent"
and Event1="PushEvent"
and Event2="CreateEvent"

Group by repository_owner

7秒......

如果事件不在＆＃34;背靠背和＃34; （参考约旦的评论），需要让它更复杂一些：

    Select repository_owner from
(
Select repository_owner,Event0,Event1,
Lead(Event0,1) OVER (Partition by repository_owner order by ts) as Event2,
Lead(Event1,1) OVER (Partition by repository_owner order by ts) as Event3,
FROM
(Select * from
(Select repository_owner,type as Event0,ts, 
LEAD(x,1) OVER(Partition by repository_owner order by ts) as Event1, 
FROM
(
SELECT repository_owner as user,created_at as ts,type as x
from [publicdata:samples.github_timeline]
where type in ("WatchEvent","PushEvent","CreateEvent")
))
where (Event0="WatchEvent" and
     Event1 in("PushEvent" ,"CreateEvent"))
OR ( Event1="CreateEvent" and
   Event0 in("PushEvent" ,"WatchEvent")))
)
   Where Event0="WatchEvent" and 
      (Event1="PushEvent" Or Event2="PushEvent") and
      Event3="CreateEvent"
Group by repository_owner

如果您的数据集太大，则会遇到此问题：Parallelizable OVER EACH BY

希望有所帮助

Answer 3

如果在非分组查询中使用“limit 100”，orchestrator将在获取前100个数据行后中断执行。

“按用户限制分组100”要求在分组之前必须计算所有数据行。然后执行分组。最后，“限制100”终于生效了。

分组来获取Google BigQuery中的不同行需要更长的时间

3 个答案: