Heroku Postgres:Out of Memory Error - 根据40的要求失败

时间:2016-04-29 13:01:07

标签: postgresql heroku out-of-memory

我尝试使用Heroku Postgres Standard0计划在Heroku中运行繁重的查询。

在远程数据库的克隆上,查询在本地运行正常。 我的本地Postgres配置的work_mem为4mb,默认值为。

数据库非常大:差不多9 GB。

以下是查询:

WITH 
AVG AS (
SELECT application_id,country_id,collection_id,category_id,COUNT(id) AS number_appearances,AVG(application_history.ranking) AS avg_ranking 
FROM application_history
WHERE application_history.date::date BETWEEN date (CURRENT_DATE - INTERVAL '1 month') AND CURRENT_DATE
GROUP BY application_id,country_id,collection_id,category_id
ORDER BY application_id ASC
),
OLD AS (
SELECT application_id AS appIdOld, country_id,collection_id,category_id,ranking AS old_ranking
FROM application_history
WHERE date::date = date_trunc('day', NOW() - interval '1 month')
),
NEW AS(
SELECT application_id AS appIdNew, country_id,collection_id,category_id,ranking AS todays_ranking 
FROM application_history
WHERE date::date = date_trunc('day',NOW())
)
SELECT AVG.application_id,AVG.country_id,AVG.collection_id,AVG.category_id,number_appearances,avg_ranking,old_ranking,todays_ranking, 
SUM(old_ranking-avg_ranking) AS delta_ranking
FROM AVG
JOIN OLD 
ON AVG.application_id = OLD.appIdOld 
AND AVG.country_id = OLD.country_id
AND AVG.collection_id = OLD.collection_id 
AND AVG.category_id = OLD.category_id
JOIN NEW
ON AVG.application_id = NEW.appIdNew 
AND AVG.country_id = NEW.country_id
AND AVG.collection_id = NEW.collection_id 
AND AVG.category_id = NEW.category_id
GROUP BY avg.application_id,avg.country_id,avg.collection_id,avg.category_id,avg.number_appearances,avg.avg_ranking,old.old_ranking,
new.todays_ranking

我试图检查单独的请求是否有效,并发现唯一的失败查询是第一个:

SELECT application_id,country_id,collection_id,category_id,COUNT(id) AS number_appearances,AVG(application_history.ranking) AS avg_ranking 
FROM application_history
WHERE application_history.date::date BETWEEN date (CURRENT_DATE - INTERVAL '1 month') AND CURRENT_DATE
GROUP BY application_id,country_id,collection_id,category_id
ORDER BY application_id ASC

每当我尝试运行它时,我都会在heroku日志中收到此错误:

2016-04-29T12:25:13Z app[postgres.15]: [DATABASE] checkpoint starting: time
2016-04-29T12:25:13Z app[postgres.15]: [DATABASE] checkpoint complete: wrote 0 buffers (0.0%); 0 transaction log file(s) added, 0 removed, 10 recycled; write=0.001 s, sync=0.000 s, total=0.026 s; sync files=0, longest=0.000 s, average=0.000 s; distance=163840 kB, estimate=225537 kB
2016-04-29T12:24:47+00:00 app[heroku-postgres]: source=DATABASE sample#current_transaction=106252061.0 sample#db_size=9551544492.0bytes sample#tables=8 sample#active-connections=3 sample#waiting-connections=0 sample#index-cache-hit-rate=0.99985 sample#table-cache-hit-rate=0.94637 sample#load-avg-1m=0.38 sample#load-avg-5m=0.49 sample#load-avg-15m=0.26 sample#read-iops=1809.2 sample#write-iops=0 sample#memory-total=3786352.0kB sample#memory-free=142060kB sample#memory-cached=3482828.0kB sample#memory-postgres=21940kB
2016-04-29T12:26:23+00:00 app[heroku-postgres]: source=DATABASE sample#current_transaction=106252061.0 sample#db_size=9551544492.0bytes sample#tables=8 sample#active-connections=4 sample#waiting-connections=0 sample#index-cache-hit-rate=0.99985 sample#table-cache-hit-rate=0.94637 sample#load-avg-1m=0.515 sample#load-avg-5m=0.495 sample#load-avg-15m=0.27 sample#read-iops=1572.7 sample#write-iops=0.010417 sample#memory-total=3786352.0kB sample#memory-free=401448kB sample#memory-cached=2274420.0kB sample#memory-postgres=972772kB
2016-04-29T12:27:39Z app[postgres.30904]: [DATABASE] out of memory
Detail: Failed on request of size 40.
Query: SELECT application_id,country_id,collection_id,category_id,COUNT(id) AS number_appearances,AVG(application_history.ranking) AS avg_ranking 
FROM application_history
WHERE application_history.date::date BETWEEN date (CURRENT_DATE - INTERVAL '1 month') AND CURRENT_DATE
GROUP BY application_id,country_id,collection_id,category_id
ORDER BY application_id ASC

在SO上阅读了相关问题之后,我认为我的查询使用了太多内存。但是,考虑到我的工作量很低,为什么它会在当地工作呢? 我还注意到该查询甚至没有出现在Heroku的数据库指标部分,而且我的错误日志也不包含通常的Heroku error codes,例如&#39 ; H12'

更具体的问题是如何在不升级到更高计划的情况下解决这个问题?

感谢您的帮助。

编辑:

这是我在添加' EXPLAIN(ANALYZE,BUFFERS)'时返回的日志中的一行。条款,正如Richard Huxton所建议的那样。

[DATABASE] temporary file: path "pg_tblspc/16386/PG_9.5_201510051/pgsql_tmp/pgsql_tmp30904.5", size 1073741824  

创建的临时文件大约为1GB,即Standard0计划的缓存限制。当我缩小查询的结果时,我强烈怀疑我达到了计划限制。

编辑2:

我确实达到了Heroku的内存限制。因此我搬到了Digital Ocean& Dokku能够根据需要调整postgres和节点。

0 个答案:

没有答案