Elasticsearch Kubernetes Pod失败并崩溃-重启重启失败的容器

时间:2019-06-16 20:48:09

标签: elasticsearch kubernetes kubernetes-helm kubernetes-pod

我正在使用这些步骤https://vocon-it.com/2019/03/04/kubernetes-9-installing-elasticsearch-using-helm-charts/来使用头盔图表安装elasticsearch。 我正在使用这些elasticsearch掌舵图-https://github.com/helm/charts/tree/master/stable/elasticsearch

ade-db2-6c56bc6dfd-jfnw4                                       0/1     ImagePullBackOff        0          30m
augmented-data-explorer-759b9bd96-jds24                        1/1     Running                 0          30m
augmented-data-explorer-elasticsearch-client-9f8c7984b-gz4kd   0/1     Init:CrashLoopBackOff   10         30m
augmented-data-explorer-elasticsearch-data-0                   0/1     Init:CrashLoopBackOff   10         30m
augmented-data-explorer-elasticsearch-master-0                 0/1     Init:CrashLoopBackOff   10         30m
 [root@dv-demo4-master-1 ~]# kubectl -n zen describe pod augmented-data-explorer-elasticsearch-data 
Name:               augmented-data-explorer-elasticsearch-data-0
Namespace:          zen
Priority:           0
PriorityClassName:  <none>
Node:               172.16.196.167/172.16.196.167
Start Time:         Sun, 16 Jun 2019 10:31:56 -0700
Labels:             app=elasticsearch
                    component=data
                    controller-revision-hash=augmented-data-explorer-elasticsearch-data-7fbd495c9f
                    release=augmented-data-explorer
                    role=data
                    statefulset.kubernetes.io/pod-name=augmented-data-explorer-elasticsearch-data-0
Annotations:        kubernetes.io/psp: augmented-data-explorer-elasticsearch
Status:             Pending
IP:                 10.1.213.210
Controlled By:      StatefulSet/augmented-data-explorer-elasticsearch-data
Init Containers:
  sysctl:
    Container ID:  docker://42be82c2aedb8971383b1d0ce9aa23f65b63294be1f4b1dd2addffa13f2173b3
    Image:         busybox:latest
    Image ID:      docker-pullable://busybox@sha256:7a4d4ed96e15d6a3fe8bfedb88e95b153b93e230a96906910d57fc4a13210160
    Port:          <none>
    Host Port:     <none>
    Command:
      sysctl
      -w
      vm.max_map_count=262144
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sun, 16 Jun 2019 16:43:18 -0700
      Finished:     Sun, 16 Jun 2019 16:43:18 -0700
    Ready:          False
    Restart Count:  77
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from augmented-data-explorer-elasticsearch-data-token-mqtb5 (ro)
  chown:
    Container ID:  
    Image:         docker.elastic.co/elasticsearch/elasticsearch-oss:6.7.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -c
      set -e; set -x; chown elasticsearch:elasticsearch /usr/share/elasticsearch/data; for datadir in $(find /usr/share/elasticsearch/data -mindepth 1 -maxdepth 1 -not -name ".snapshot"); do
        chown -R elasticsearch:elasticsearch $datadir;
      done; chown elasticsearch:elasticsearch /usr/share/elasticsearch/logs; for logfile in $(find /usr/share/elasticsearch/logs -mindepth 1 -maxdepth 1 -not -name ".snapshot"); do
        chown -R elasticsearch:elasticsearch $logfile;
      done

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from augmented-data-explorer-elasticsearch-data-token-mqtb5 (ro)
Containers:
  elasticsearch:
    Container ID:   
    Image:          docker.elastic.co/elasticsearch/elasticsearch-oss:6.7.0
    Image ID:       
    Port:           9300/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:  1
    Requests:
      cpu:      25m
      memory:   1536Mi
    Readiness:  http-get http://:9200/_cluster/health%3Flocal=true delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DISCOVERY_SERVICE:           augmented-data-explorer-elasticsearch-discovery
      NODE_MASTER:                 false
      PROCESSORS:                  1 (limits.cpu)
      ES_JAVA_OPTS:                -Djava.net.preferIPv4Stack=true -Xms1536m -Xmx1536m  
      EXPECTED_MASTER_NODES:       1
      MINIMUM_MASTER_NODES:        1
      RECOVER_AFTER_MASTER_NODES:  1
      bootstrap.memory_lock:       true
    Mounts:
      /post-start-hook.sh from config (rw)
      /pre-stop-hook.sh from config (rw)
      /usr/share/elasticsearch/config/elasticsearch.yml from config (rw)
      /usr/share/elasticsearch/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from augmented-data-explorer-elasticsearch-data-token-mqtb5 (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-augmented-data-explorer-elasticsearch-data-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      augmented-data-explorer-elasticsearch
    Optional:  false
  augmented-data-explorer-elasticsearch-data-token-mqtb5:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  augmented-data-explorer-elasticsearch-data-token-mqtb5
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                      From                     Message
  ----     ------   ----                     ----                     -------
  Normal   Pulled   42m (x70 over 6h12m)     kubelet, 172.16.196.167  Successfully pulled image "busybox:latest"
  Normal   Pulling  37m (x71 over 6h12m)     kubelet, 172.16.196.167  pulling image "busybox:latest"
  Warning  BackOff  117s (x1713 over 6h12m)  kubelet, 172.16.196.167  Back-off restarting failed container

pvc已绑定-kubectl -n zen get pvc

data-augmented-data-explorer-elasticsearch-data-0     Bound    pvc-97fab8fa-905c-11e9-9850-00163e01c3eb   30Gi       RWO            oketi-gluster   176m
data-augmented-data-explorer-elasticsearch-master-0   Bound    pvc-97fbb612-905c-11e9-9850-00163e01c3eb   4Gi        RWO            oketi-gluster   176m

这是我用于Elasticsearch-data的PVC

kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: {{ template "elasticsearch.name" . }}
    component: data
    release: {{ .Release.Name }}
  name: {{ .Values.adeElasticSearchDataPVC.name }}
  namespace: {{ toYaml .Values.namespace }}
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: oketi-gluster
  resources:
    requests:
      storage: 30Gi
  volumeMode: Filesystem

这是用于Elasticsearch-master的PVC

kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: {{ template "elasticsearch.name" . }}
    component: master
    release: {{ .Release.Name }}
    role: master
  name: {{ .Values.adeElasticSearchMasterPVC.name }}
  namespace:  {{ toYaml .Values.namespace }}
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: oketi-gluster
  resources:
    requests:
      storage: 4Gi
  volumeMode: Filesystem

我在日志中找不到有用的东西

kubectl -n zen log augmented-data-explorer-elasticsearch-client-9f8c7984b-gz4kd
Error from server (BadRequest): container "elasticsearch" in pod "augmented-data-explorer-elasticsearch-client-9f8c7984b-gz4kd" is waiting to start: PodInitializing

关于如何解决此问题或如何调试它的任何想法?

1 个答案:

答案 0 :(得分:0)

就像@Nidhi在评论中提到的那样,通过调整相关ES容器内vm.max_map_count中特定的/etc/sysctl.conf虚拟内存限制(源文档here)已解决了自举Elasticserach容器的问题,并使用mlockall函数通过授予ES用户锁定内存权限来防止内存分配被换出。

可以通过修改/etc/security/limits.conf来进行调整,例如:

elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited