其中一个容器因系统日志中出现以下错误消息而被挂起并且没有响应。我怎样才能找出问题所在？
sys日志错误：
Jul 20 12:58:26 B2BTestServer dockerd[1140]: time=“2017-07-20T12:58:26.879111489Z” level=error msg=“containerd: deleting container” error=“exit status 1: “container 5f090b50b3dd7840dd296bd3eede2c4db171b5787944317f0a0e52d71c368361 does not exist\none or more of the container deletions failed\n””
Jul 20 13:30:01 B2BTestServer CRON[42898]: (xxxx) CMD (docker exec xxApp_worker_1 ./artisan omni:key b2b_fwad login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42899]: (xxxx) CMD (docker exec xxApp_app_1 ./artisan omni:key b2b_fwad login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42901]: (xxxx) CMD (docker exec xxApp_worker_1 ./artisan omni:key b2b_yww login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer CRON[42900]: (xxxx) CMD (docker exec xxApp_app_1 ./artisan omni:key b2b_yww login >/dev/null 2>&1)
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.382668472Z” level=error msg="Error setting up exec command in container xxApp_app_1: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.382714110Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/exec returned error: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.392014905Z” level=error msg="Error setting up exec command in container xxApp_app_1: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.392237089Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/exec returned error: Container f743904a3558b5475df8e574e9b27a5ca8b2c7256da59c356f4da1a9bda72e4b is not running"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.429292849Z” level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n"
Jul 20 13:30:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:01.463663587Z” level=error msg=“Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n”
Jul 20 13:30:36 B2BTestServer kernel: [262136.359656] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262136.583654] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.120768757Z” level=error msg=“containerd: deleting container” error="exit status 1: “container f80c3f7d41bcbb2133036c795068ea2200b8043bce1a043f5fb5e09d09c2f720 does not exist\none or more of the container deletions failed\n”"
Jul 20 13:30:37 B2BTestServer kernel: [262136.651251] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.668608] vethebfd8d9: renamed from eth0
Jul 20 13:30:37 B2BTestServer kernel: [262136.716289] aufs au_opts_verify:1597:dockerd[2634]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262136.738679] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.753388] device vethe12115e left promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262136.762750] br-e80847589bfa: port 2(vethe12115e) entered disabled state
Jul 20 13:30:37 B2BTestServer kernel: [262136.804865] device vethdb0a6ac entered promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262136.814982] IPv6: ADDRCONF(NETDEV_UP): vethdb0a6ac: link is not ready
Jul 20 13:30:37 B2BTestServer kernel: [262136.827291] br-e80847589bfa: port 1(vethdb0a6ac) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262136.840256] br-e80847589bfa: port 1(vethdb0a6ac) entered forwarding state
Jul 20 13:30:37 B2BTestServer systemd-udevd[43041]: Could not generate persistent MAC address for vethdb0a6ac: No such file or directory
Jul 20 13:30:37 B2BTestServer systemd-udevd[43040]: Could not generate persistent MAC address for vethf7a98cb: No such file or directory
Jul 20 13:30:37 B2BTestServer kernel: [262137.026630] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579534632Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579622593Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.579632100Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:37 B2BTestServer kernel: [262137.101169] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37Z” level=info msg="Firewalld running: false"
Jul 20 13:30:37 B2BTestServer kernel: [262137.207004] aufs au_opts_verify:1597:dockerd[25633]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:37 B2BTestServer kernel: [262137.248708] eth0: renamed from vethf7a98cb
Jul 20 13:30:37 B2BTestServer systemd-udevd[43171]: Could not generate persistent MAC address for veth91e8259: No such file or directory
Jul 20 13:30:37 B2BTestServer systemd-udevd[43172]: Could not generate persistent MAC address for veth3fff19d: No such file or directory
Jul 20 13:30:37 B2BTestServer kernel: [262137.273642] device veth3fff19d entered promiscuous mode
Jul 20 13:30:37 B2BTestServer kernel: [262137.287783] IPv6: ADDRCONF(NETDEV_UP): veth3fff19d: link is not ready
Jul 20 13:30:37 B2BTestServer kernel: [262137.307604] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262137.320987] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:37 B2BTestServer kernel: [262137.335059] IPv6: ADDRCONF(NETDEV_CHANGE): vethdb0a6ac: link becomes ready
Jul 20 13:30:37 B2BTestServer kernel: [262137.349736] br-e80847589bfa: port 2(veth3fff19d) entered disabled state
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883496197Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883793205Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37.883937606Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:37 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:37Z” level=info msg="Firewalld running: false"
Jul 20 13:30:38 B2BTestServer kernel: [262137.524751] eth0: renamed from veth91e8259
Jul 20 13:30:38 B2BTestServer kernel: [262137.544382] IPv6: ADDRCONF(NETDEV_CHANGE): veth3fff19d: link becomes ready
Jul 20 13:30:38 B2BTestServer kernel: [262137.556891] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262137.569301] br-e80847589bfa: port 2(veth3fff19d) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262137.704615] aufs au_opts_verify:1597:dockerd[3229]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.803344] aufs au_opts_verify:1597:dockerd[3229]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.902578] aufs au_opts_verify:1597:dockerd[1209]: dirperm1 breaks the protection by the permission bits on the lower branch
Jul 20 13:30:38 B2BTestServer kernel: [262137.926880] device veth4ff9aca entered promiscuous mode
Jul 20 13:30:38 B2BTestServer systemd-udevd[43302]: Could not generate persistent MAC address for veth4ff9aca: No such file or directory
Jul 20 13:30:38 B2BTestServer kernel: [262137.938135] IPv6: ADDRCONF(NETDEV_UP): veth4ff9aca: link is not ready
Jul 20 13:30:38 B2BTestServer systemd-udevd[43301]: Could not generate persistent MAC address for vetha4d818b: No such file or directory
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495129396Z” level=warning msg="Your kernel does not support swap memory limit."
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495215954Z” level=warning msg="Your kernel does not support cgroup rt period"
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38.495226161Z” level=warning msg="Your kernel does not support cgroup rt runtime"
Jul 20 13:30:38 B2BTestServer dockerd[1140]: time=“2017-07-20T13:30:38Z” level=info msg="Firewalld running: false"
Jul 20 13:30:38 B2BTestServer kernel: [262138.116861] eth0: renamed from vetha4d818b
Jul 20 13:30:38 B2BTestServer kernel: [262138.136455] IPv6: ADDRCONF(NETDEV_CHANGE): veth4ff9aca: link becomes ready
Jul 20 13:30:38 B2BTestServer kernel: [262138.149540] br-e80847589bfa: port 4(veth4ff9aca) entered forwarding state
Jul 20 13:30:38 B2BTestServer kernel: [262138.162430] br-e80847589bfa: port 4(veth4ff9aca) entered forwarding state
Jul 20 13:35:31 B2BTestServer dockerd[1140]: time=“2017-07-20T13:35:31.489837293Z” level=error msg="Handler for POST /v1.25/containers/xxApp_app_1/start returned error: Container already started"
Jul 20 13:36:05 B2BTestServer dockerd[1140]: time=“2017-07-20T13:36:05.230362142Z” level=error msg=“Handler for POST /v1.25/containers/xxApp_worker_1/start returned error: Container already started”
Jul 20 13:50:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:50:01.606622410Z” level=error msg="Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n"
Jul 20 13:50:01 B2BTestServer dockerd[1140]: time=“2017-07-20T13:50:01.687809674Z” level=error msg=“Error running exec in container: rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused “process_linux.go:83: executing setns process caused \“exit status 15\””\n”
以下是Docker版本信息：
sudo docker info
Containers: 8
Running: 3
Paused: 0
Stopped: 5
Images: 14
Server Version: 1.13.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 126
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-83-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.804 GiB
Name: TestServer
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false