kubelet不会为/ var / log / containers创建符号链接

Question

我正在尝试使用ansible repo在我的k8s群集上设置EFK堆栈。

当我尝试浏览kibana仪表板时，它会向我显示下一个输出：

经过一些研究后，我发现我没有Fluentd检测到的任何日志。 我在minions上运行k8s 1.2.4，在master上运行1.2.0。 我成功理解的是，kubelet创建/ var / log / containers目录，并将集群中运行的所有容器的符号链接放入其中。之后，Fluentd从minion安装share / var / log卷，最终访问所有日志容器。因此，它可以将这些日志发送到弹性搜索。

在我的情况下，我创建了/ var / log / containers，但它是空的，即使/ var / lib / docker / containers也不包含任何日志文件。 我以前使用以下控制器和服务进行EFK堆栈设置：

ES-controller.yaml

apiVersion: v1 kind: ReplicationController metadata: name: elasticsearch-logging-v1 namespace: kube-system labels: k8s-app: elasticsearch-logging version: v1 kubernetes.io/cluster-service: "true" spec: replicas: 2 selector: k8s-app: elasticsearch-logging version: v1 template: metadata: labels: k8s-app: elasticsearch-logging version: v1 kubernetes.io/cluster-service: "true" spec: containers: - image: gcr.io/google_containers/elasticsearch:v2.4.1 name: elasticsearch-logging resources: # need more cpu upon initialization, therefore burstable class limits: cpu: 1000m requests: cpu: 100m ports: - containerPort: 9200 name: db protocol: TCP - containerPort: 9300 name: transport protocol: TCP volumeMounts: - name: es-persistent-storage mountPath: /data env: - name: "NAMESPACE" valueFrom: fieldRef: fieldPath: metadata.namespace volumes: - name: es-persistent-storage emptyDir: {}

ES-service.yaml

apiVersion: v1 kind: Service metadata: name: elasticsearch-logging namespace: kube-system labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" kubernetes.io/name: "Elasticsearch" spec: ports: - port: 9200 protocol: TCP targetPort: db selector: k8s-app: elasticsearch-logging

fluentd-es.yaml

apiVersion: v1 kind: Pod metadata: name: fluentd-es-v1.20 namespace: kube-system labels: k8s-app: fluentd-es version: v1.20 spec: containers: - name: fluentd-es image: gcr.io/google_containers/fluentd-elasticsearch:1.20 command: - '/bin/sh' - '-c' - '/usr/sbin/td-agent 2>&1 >> /var/log/fluentd.log' resources: limits: cpu: 100m volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers

kibana-controller.yaml

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: kibana-logging namespace: kube-system labels: k8s-app: kibana-logging kubernetes.io/cluster-service: "true" spec: replicas: 1 selector: matchLabels: k8s-app: kibana-logging template: metadata: labels: k8s-app: kibana-logging spec: containers: - name: kibana-logging image: gcr.io/google_containers/kibana:v4.6.1 resources: # keep request = limit to keep this container in guaranteed class limits: cpu: 100m requests: cpu: 100m env: - name: "ELASTICSEARCH_URL" value: "http://elasticsearch-logging:9200" ports: - containerPort: 5601 name: ui protocol: TCP

kibana-service.yaml

apiVersion: v1 kind: Service metadata: name: kibana-logging namespace: kube-system labels: k8s-app: kibana-logging kubernetes.io/cluster-service: "true" kubernetes.io/name: "Kibana" spec: type: NodePort ports: - port: 5601 protocol: TCP targetPort: ui selector: k8s-app: kibana-logging

更新

我改变了fluentd-es.yaml如下：

apiVersion: v1 kind: Pod metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: fluentd-logging spec: containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.15 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers

但是当我运行一个“命名网关”的pod时，我在流畅的日志中得到了下一个错误：  /var/log/containers/gateway-c3cuu_default_gateway-d5966a86e7cb1519329272a0b900182be81f55524227db2f524e6e23cd75ba04.log unreadable. It is excluded and would be examined next time.

Answer 1

最后我发现导致这个问题的原因。 从CentOS 7 repo安装docker时，有一个选项（--log-driver = journald）强制docker将日志输出运行到journald。默认行为是将这些日志写入json.log文件。所以，我唯一要做的就是删除/ etc / sysconfig / docker中最后提到的选项。