Question

我已经将我的AKS Azure Kubernetes集群更新为版本1.11.5，在该集群中正在运行MongoDB Statefulset：

状态集是使用以下文件创建的：

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: default-view
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default
---
apiVersion: v1
kind: Service
metadata:
  name: mongo
  labels:
    name: mongo
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 2
  template:
    metadata:
      labels:
        role: mongo
        environment: test
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--bind_ip"
            - 0.0.0.0            
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"
  volumeClaimTemplates:
  - metadata:
      name: mongo-persistent-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "managed-premium"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 32Gi

在将集群更新为新的k8s版本后，出现此错误：

mongo-0                        1/2     CrashLoopBackOff   6          9m
mongo-1                        2/2     Running            0          1h

窗格中的详细日志如下：

2018-12-18T14:28:44.281+0000 W STORAGE  [initandlisten] Detected configuration for non-active storage engine mmapv1 when current storage engine is wiredTiger
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten]
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten]
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten]
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2018-12-18T14:28:44.281+0000 I CONTROL  [initandlisten]
2018-12-18T14:28:44.477+0000 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/data/db/diagnostic.data'
2018-12-18T14:28:44.478+0000 I REPL     [initandlisten] Rollback ID is 7
2018-12-18T14:28:44.479+0000 I REPL     [initandlisten] Recovering from stable timestamp: Timestamp(1545077719, 1) (top of oplog: { ts: Timestamp(1545077349, 1), t: 5 }, appliedThrough: { ts: Timestamp(1545077719, 1), t: 6 }, TruncateAfter: Timestamp(0, 0))
2018-12-18T14:28:44.480+0000 I REPL     [initandlisten] Starting recovery oplog application at the stable timestamp: Timestamp(1545077719, 1)
2018-12-18T14:28:44.480+0000 F REPL     [initandlisten] Applied op { : Timestamp(1545077719, 1) } not found. Top of oplog is { : Timestamp(1545077349, 1) }.
2018-12-18T14:28:44.480+0000 F -        [initandlisten] Fatal Assertion 40313 at src/mongo/db/repl/replication_recovery.cpp 361
2018-12-18T14:28:44.480+0000 F -        [initandlisten]

***aborting after fassert() failure

看来这两个实例不同步，无法恢复。有人可以帮忙吗？

Answer 1

我有解决此问题的方法：

将MongoDB容器添加到集群以转储和还原MongoDB数据
转储当前数据库
删除MongoDB实例
重新创建一个新的MongoDB实例
将数据恢复到新实例

很遗憾，这会带来停机时间

Answer 2

寻找可靠和/或官方来源的答案。

一个官方消息来源是“ Running MongoDB on Kubernetes with StatefulSets”（从2017年开始，因此可能需要一些适应/发展），但是您似乎已经遵循了。

您的错误消息是2个月前在mongodb.org SERVER 37724中看到的

在4.0中，我们确实对日志记录过程进行了更改，在日志记录过程中，它遵循操作日志而不是数据文件本身。有机会成为这里正在发生的事情。

要进行测试，请尝试使用MongoDB 3.6，以查看问题是否仍然存在。

kubernetes更新后，mongodb StatefulSet在kubernetes上不再工作

2 个答案: