Question

我过去15天一直在使用Prometheus。从昨天开始，该服务由于内存不足而失败。

我已将内存从4GB增加到8GB，但现在无法启动Prometheus服务，并失败并显示以下错误：

Nov 15 10:25:20 prometheus: level=info ts=2019-11-15T18:25:20.393Z caller=head.go:598 component=tsdb msg="head GC completed" duration=22.375774ms
Nov 15 10:25:35 prometheus: level=info ts=2019-11-15T18:25:35.135Z caller=head.go:668 component=tsdb msg="WAL checkpoint complete" first=549 last=550 duration=14.741502221s
Nov 15 10:25:35 prometheus: level=info ts=2019-11-15T18:25:35.795Z caller=head.go:598 component=tsdb msg="head GC completed" duration=22.593533ms
Nov 15 10:25:36 prometheus: level=info ts=2019-11-15T18:25:36.796Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50E9ZP9F1ZSY2YD30GEHK duration=732.572029ms
Nov 15 10:25:37 prometheus: level=error ts=2019-11-15T18:25:37.062Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50E9ZP9F1ZSY2YD30GEHK:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:38 prometheus: level=info ts=2019-11-15T18:25:38.824Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50G8EYTYBPWWHWZ1RJV5A duration=761.592662ms
Nov 15 10:25:39 prometheus: level=error ts=2019-11-15T18:25:39.087Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50G8EYTYBPWWHWZ1RJV5A:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:41 prometheus: level=info ts=2019-11-15T18:25:41.816Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50K6ZJFJQ4217JW2E5XBV duration=729.049353ms
Nov 15 10:25:42 prometheus: level=error ts=2019-11-15T18:25:42.087Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50K6ZJFJQ4217JW2E5XBV:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:46 prometheus: level=info ts=2019-11-15T18:25:46.814Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR50R37245ZE56EZNH0335G duration=727.559705ms
Nov 15 10:25:47 prometheus: level=error ts=2019-11-15T18:25:47.084Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR50R37245ZE56EZNH0335G:mmap files: mmap: cannot allocate memory]"
Nov 15 10:25:55 prometheus: level=info ts=2019-11-15T18:25:55.807Z caller=compact.go:496 component=tsdb msg="write block" mint=1573826400000 maxt=1573833600000 ulid=01DSR510WDE18F41GJBRVXBEDE duration=722.624112ms
Nov 15 10:25:55 prometheus: level=error ts=2019-11-15T18:25:56.080Z caller=db.go:604 component=tsdb msg="compaction failed" err="reload blocks: unexpected corrupted block:map[01DSR510WDE18F41GJBRVXBEDE:mmap files: mmap: cannot allocate memory]"

普罗米修斯服务：


[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

我已经在同一台服务器上安装了influxdb grafana。

请让我知道确切的问题是什么？

预先感谢

Answer 1

您是否正在运行32位Prometheus？您可能用完了地址空间。我不认为Prometheus计划在32位系统上实现此功能，因为它{4}的所有使用mmap的情况下仅用完4GB的地址空间。

更多详情，请访问https://github.com/prometheus/prometheus/issues/4392#issuecomment-433721793

Prometheus服务无法启动

1 个答案: