我已经基于此处共享的一个查询进行了查询：https://github.com/sagishporer/big-query-queries-for-firebase/wiki/Query:-Daily-retention 计算从Firebase传输的数据在BigQuery中的用户保留率。
到目前为止，它一直在工作，但是随着数据集变得更大，由于以下错误，它不再能够运行它：
查询执行期间超出了资源：无法在分配的内存中执行查询。高峰使用：限制的129％。内存消耗大户：用于分析OVER（）子句的排序操作：100％
查询如下：
SELECT
install_date,
SUM(CASE
WHEN days_since_install = 0 THEN users
ELSE 0 END) AS day_0,
SUM(CASE
WHEN days_since_install = 1 THEN users
ELSE 0 END) AS day_1,
SUM(CASE
WHEN days_since_install = 2 THEN users
ELSE 0 END) AS day_2,
SUM(CASE
WHEN days_since_install = 3 THEN users
ELSE 0 END) AS day_3,
SUM(CASE
WHEN days_since_install = 4 THEN users
ELSE 0 END) AS day_4,
SUM(CASE
WHEN days_since_install = 5 THEN users
ELSE 0 END) AS day_5,
SUM(CASE
WHEN days_since_install = 6 THEN users
ELSE 0 END) AS day_6,
SUM(CASE
WHEN days_since_install = 7 THEN users
ELSE 0 END) AS day_7,
SUM(CASE
WHEN days_since_install = 8 THEN users
ELSE 0 END) AS day_8,
SUM(CASE
WHEN days_since_install = 9 THEN users
ELSE 0 END) AS day_9,
SUM(CASE
WHEN days_since_install = 10 THEN users
ELSE 0 END) AS day_10,
SUM(CASE
WHEN days_since_install = 11 THEN users
ELSE 0 END) AS day_11,
SUM(CASE
WHEN days_since_install = 12 THEN users
ELSE 0 END) AS day_12,
SUM(CASE
WHEN days_since_install = 13 THEN users
ELSE 0 END) AS day_13,
SUM(CASE
WHEN days_since_install = 14 THEN users
ELSE 0 END) AS day_14,
SUM(CASE
WHEN days_since_install = 15 THEN users
ELSE 0 END) AS day_15,
SUM(CASE
WHEN days_since_install = 16 THEN users
ELSE 0 END) AS day_16,
SUM(CASE
WHEN days_since_install = 17 THEN users
ELSE 0 END) AS day_17,
SUM(CASE
WHEN days_since_install = 18 THEN users
ELSE 0 END) AS day_18,
SUM(CASE
WHEN days_since_install = 19 THEN users
ELSE 0 END) AS day_19,
SUM(CASE
WHEN days_since_install = 20 THEN users
ELSE 0 END) AS day_20,
SUM(CASE
WHEN days_since_install = 21 THEN users
ELSE 0 END) AS day_21,
SUM(CASE
WHEN days_since_install = 22 THEN users
ELSE 0 END) AS day_22,
SUM(CASE
WHEN days_since_install = 23 THEN users
ELSE 0 END) AS day_23,
SUM(CASE
WHEN days_since_install = 24 THEN users
ELSE 0 END) AS day_24,
SUM(CASE
WHEN days_since_install = 25 THEN users
ELSE 0 END) AS day_25,
SUM(CASE
WHEN days_since_install = 26 THEN users
ELSE 0 END) AS day_26,
SUM(CASE
WHEN days_since_install = 27 THEN users
ELSE 0 END) AS day_27,
SUM(CASE
WHEN days_since_install = 28 THEN users
ELSE 0 END) AS day_28,
SUM(CASE
WHEN days_since_install = 29 THEN users
ELSE 0 END) AS day_29,
SUM(CASE
WHEN days_since_install = 30 THEN users
ELSE 0 END) AS day_30
FROM (
SELECT
DATE(TIMESTAMP_MICROS(user_first_touch_timestamp)) AS install_date,
DATE(TIMESTAMP_MICROS(event_timestamp)) AS event_realdate,
DATE_DIFF(DATE(TIMESTAMP_MICROS(event_timestamp)), DATE(TIMESTAMP_MICROS(user_first_touch_timestamp)), day) AS days_since_install,
COUNT(DISTINCT user_pseudo_id) AS users
FROM
`dataset.events_2019*`
WHERE
event_name = 'user_engagement'
AND user_pseudo_id NOT IN (
SELECT
user_pseudo_id
FROM (
SELECT
MIN(global_session_id),
user_pseudo_id,
user_first_touch_timestamp,
event_timestamp
FROM (
SELECT
*,
IF (previous_event='some_event'
AND LAG(global_session_id,1)OVER (ORDER BY global_session_id, event_name)=global_session_id,
LAG(global_session_id,1) OVER (ORDER BY global_session_id, event_name),
NULL) AS match
FROM (
SELECT
*,
LAG(event_name,1) OVER (ORDER BY global_session_id, event_name) AS previous_event
FROM (
SELECT
global_session_id,
event_name,
user_first_touch_timestamp,
event_timestamp,
user_pseudo_id
FROM (
SELECT
global_session_id,
event_name,
user_pseudo_id,
event_timestamp,
user_first_touch_timestamp,
IF (some_kill=1,
global_session_id,
NULL) AS session_some_kill,
IF (event_name='user_engagement',
global_session_id,
NULL) AS session
FROM (
SELECT
*,
CASE
WHEN event_params.key = 'Kills' AND event_params.value.int_value>0 THEN 1
ELSE 0
END AS some_kill,
SUM(is_new_session) OVER (ORDER BY user_pseudo_id, event_timestamp, event_name) AS global_session_id,
SUM(is_new_session) OVER (PARTITION BY user_pseudo_id ORDER BY event_timestamp) AS user_session_id
FROM (
SELECT
*,
CASE
WHEN event_timestamp - last_event >= (30 * 60 * 1000) OR last_event IS NULL THEN 1
ELSE 0
END AS is_new_session
FROM (
SELECT
user_pseudo_id,
event_timestamp,
event_name,
event_params,
user_first_touch_timestamp,
LAG(event_timestamp,1) OVER (PARTITION BY user_pseudo_id ORDER BY event_timestamp) AS last_event
FROM (
SELECT
user_pseudo_id,
event_timestamp,
event_name,
event_params,
user_first_touch_timestamp
FROM `dataset.events_2019*`,
UNNEST (event_params) AS event_params)
) last
) agg
)
)
WHERE
session_some_kill IS NOT NULL
OR session IS NOT NULL
GROUP BY
global_session_id,
event_name,
user_first_touch_timestamp,
event_timestamp,
user_pseudo_id
ORDER BY
global_session_id ) ) )
WHERE
match IS NOT NULL
AND event_timestamp-user_first_touch_timestamp<1.8e+9
GROUP BY
user_pseudo_id,
user_first_touch_timestamp,
event_timestamp))
GROUP BY
install_date,
event_realdate,
days_since_install )
GROUP BY
install_date
HAVING
day_0 > 0 /* Remove older dates - not enough data, you should also ignore the first record for partial data */
ORDER BY
install_date
答案 0 :(得分：0)
我们通过2次传球来做这种事情。首先，我们计算每个用户的活动天数，然后进行所需的任何计算。
我们为每个用户使用类似的方法来存储活动日：
$b1 = Get-Content $file
$b2 = $b1 | Select-Object -Skip 1
活动天数存储为一个位域，每个INT64最多可存储64天，而不是分别计算和存储每个活动天数。 您可以根据需要增加更多的天数，即通过INT64偏移增加64天。 查询运行和导出非常快。
这是UTC，您可以根据需要将其转换为本地时间。
我们只需要最大的一次，因为我们通过user_id分组并使用帐户链接，并且当用户卸载，重新安装和链接时，该用户将获得另一个first_touch_timestamp，它比旧事件要新。
希望这会有所帮助，