MongoDB在生产环境中因WT事务缓慢而崩溃

Question

问题行为

• MongoDB在生产环境中经常崩溃。至少每天一次。在每次崩溃之前，只有事情记录在日志中，如下所示：“慢速WT事务”。

在崩溃[grep]之前怀疑日志：

2018-12-05T16:59:02.569+0000 D STORAGE  [conn1170] Slow WT transaction. Lifetime of SnapshotId 1356150 was 154ms
    2018-12-05T16:59:38.214+0000 D STORAGE  [conn4] Slow WT transaction. Lifetime of SnapshotId 1357284 was 154ms
    2018-12-05T16:59:38.390+0000 D STORAGE  [conn4] Slow WT transaction. Lifetime of SnapshotId 1357291 was 103ms
    2018-12-05T17:00:03.076+0000 D STORAGE  [conn1170] Slow WT transaction. Lifetime of SnapshotId 1357925 was 158ms
    2018-12-05T17:00:24.940+0000 D STORAGE  [conn4] Slow WT transaction. Lifetime of SnapshotId 1358710 was 115ms
    2018-12-05T17:00:46.096+0000 D STORAGE  [conn8] Slow WT transaction. Lifetime of SnapshotId 1359381 was 105ms
    2018-12-05T17:00:59.304+0000 D STORAGE  [conn4] Slow WT transaction. Lifetime of SnapshotId 1359903 was 142ms
    2018-12-05T17:01:13.309+0000 D STORAGE  [conn26] Slow WT transaction. Lifetime of SnapshotId 1360311 was 164ms
    2018-12-05T17:01:13.554+0000 D STORAGE  [conn26] Slow WT transaction. Lifetime of SnapshotId 1360362 was 143ms
    2018-12-05T17:01:23.541+0000 D STORAGE  [conn4] Slow WT transaction. Lifetime of SnapshotId 1360671 was 116ms
    2018-12-05T17:01:23.761+0000 D STORAGE  [conn4] Slow WT transaction. Lifetime of SnapshotId 1360679 was 110ms
    2018-12-05T17:01:29.340+0000 D STORAGE  [conn4] Slow WT transaction. Lifetime of SnapshotId 1360880 was 203ms

环境：操作系统+浏览器/客户端+版本

MongoDB版本信息：

db version v3.6.8
git version: 6bc9ed599c3fa164703346a22bad17e33fa913e4
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
    distmod: rhel70
    distarch: x86_64
    target_arch: x86_64

操作系统信息：

NAME=CentOS Linux
VERSION=7 (Core)
ID=centos
ID_LIKE: rhel fedora
VERSION_ID=7
PRETTY_NAME=CentOS Linux 7 (Core)

下面是配置文件-

mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /apps/mongodb/data
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1  # Listen to local interface only, comment to listen on all interfaces.

#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:

可能的解决方案[已尝试]

• 尝试增加系统限制，但在2天后连续发出。

当前ulimit：

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 64045
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 128000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 64045
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

如果您需要更多信息，请告诉我。