AWS ECS容器卡在重启循环中

时间:2019-11-12 08:46:19

标签: amazon-ecs

尝试启动我的第一个ECS集群。 容器已成功启动,但每分钟左右都会重新启动。 群集服务正在使用应用程序负载平衡器。

任务定义:

{
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 80,
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "command": [
        "bash",
        "deploy.sh"
      ],
      "cpu": 512,
      "memory": 1961,
      "memoryReservation": 256,
      "image": "1",
      "essential": true,
    }
  ],
  "placementConstraints": [],
  "memory": null,
  "taskRoleArn": null,
  "compatibilities": [
    "EC2"
  ],
  "taskDefinitionArn": "arn:aws:ecs:...",
  "family": "service",
  "networkMode": "awsvpc",
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": []
}

ecs代理日志(已删除和重命名的详细信息),每分钟左右都会经历以下循环

00:05Z [INFO] Handling ENI attachment
00:05Z [INFO] Starting ENI ack timer 
00:05Z [INFO] Adding task eni attachment 
00:05Z [INFO] Emitting task ENI attached event for: ENI Attachment: 
00:05Z [INFO] TaskHandler: Adding event: TaskChange: arn:... -> NONE, ENI Attachment:  sent: false
00:05Z [INFO] TaskHandler: Sending task attachment change: TaskChange: arn:... -> NONE, ENI Attachment:  sent: false
00:11Z [INFO] Managed task arn:... unable to create state change event for container  create container state change event api: status not recognized by ECS: NONE
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: status not recognized by ECS: NONE
00:11Z [INFO] Managed task arn:... unable to create task state change event []: create task state change event api: status not recognized by ECS: NONE
00:11Z [INFO] Managed task arn:... waiting for any previous stops to complete. Sequence number: 110
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Managed task arn:... no longer waiting
00:11Z [INFO] Managed task arn:... wait over; ready to move towards status: RUNNING
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [PULLED]
00:11Z [INFO] Managed task arn:... handling container change [{PULLED { <nil> [] <nil> [] map[]  {UNKNOWN <nil> 0 }  <nil>} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: status not recognized by ECS: PULLED
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... creating container: ~internal~ecs~pause
00:11Z [INFO] Task engine arn:... created container name mapping for task:  ~internal~ecs~pause -> 
00:11Z [INFO] Creating cgroup 
00:11Z [INFO] Managed task arn:... transitioned resource [cgroup] to [CREATED]
00:11Z [INFO] Managed task arn:... got resource [cgroup] event: [CREATED]
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... created docker container for task: ~internal~ecs~pause -> ...
00:11Z [INFO] Task engine arn:... created docker container for task: ~internal~ecs~pause -> ..., took 103.387592ms
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [CREATED]
00:11Z [INFO] Managed task arn:... handling container change [{CREATED {... <nil> [] <nil> []  com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:...  com.amazonaws.ecs.task-definition-version:15]    {UNKNOWN <nil> 0 } none 0xc00011ab00} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: status not recognized by ECS: CREATED
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... starting container: ~internal~ecs~pause
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [RUNNING]
00:11Z [INFO] Managed task arn:... handling container change [{RUNNING {... <nil> [] <nil> []  com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:...  com.amazonaws.ecs.task-definition-version:15]    {UNKNOWN <nil> 0 } none 0xc00011b600} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: status not recognized by ECS: RUNNING
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... setting up container resources for container [~internal~ecs~pause]
00:11Z [INFO] Task engine arn:... started docker container for task: ~internal~ecs~pause -> ..., took 221.148618ms
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [RUNNING]
00:11Z [INFO] Managed task arn:... handling container change [{RUNNING {... <nil> [] <nil> []  com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:...  com.amazonaws.ecs.task-definition-version:15]    {UNKNOWN <nil> 0 } none 0xc0007aed00} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... redundant container state change. ~internal~ecs~pause to RUNNING, but already RUNNING
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... associated with ip address '1.2.3.4'
00:11Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [RESOURCES_PROVISIONED]
00:11Z [INFO] Managed task arn:... handling container change [{RESOURCES_PROVISIONED {... <nil> [] <nil> [] map[]  {UNKNOWN <nil> 0 }  <nil>} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:11Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: internal container: ~internal~ecs~pause
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... pulling image  for container  concurrently
00:11Z [INFO] Task engine arn:... recording timestamp for starting image pulltime:  
00:11Z [INFO] Updating container reference  in Image State - 
00:11Z [INFO] Task engine arn:... finished pulling image  for container  in 99.366882ms
00:11Z [INFO] Managed task arn:... got container  event: [PULLED]
00:11Z [INFO] Managed task arn:... handling container change [{PULLED { <nil> [] <nil> [] map[]  {UNKNOWN <nil> 0 }  <nil>} ContainerStatusChangeEvent}] for container 
00:11Z [INFO] Managed task arn:... unable to create state change event for container  create container state change event api: status not recognized by ECS: PULLED
00:11Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:11Z [INFO] Managed task arn:... waiting for event for task
00:11Z [INFO] Task engine arn:... creating container: 
00:11Z [INFO] Task engine arn:... created container name mapping for task:   -> 
00:12Z [INFO] Task engine arn:... created docker container for task:  -> ...
00:12Z [INFO] Task engine arn:... created docker container for task:  -> ..., took 81.682728ms
00:12Z [INFO] Managed task arn:... got container  event: [CREATED]
00:12Z [INFO] Managed task arn:... handling container change [{CREATED {... <nil> [] <nil> []   com.amazonaws.ecs.task-arn:arn:aws:ecs:...  com.amazonaws.ecs.task-definition-version:15]    {UNKNOWN <nil> 0 } container: 0xc00062eb00} ContainerStatusChangeEvent}] for container 
00:12Z [INFO] Managed task arn:... unable to create state change event for container  create container state change event api: status not recognized by ECS: CREATED
00:12Z [INFO] api/task: Updating task's known status to: CREATED, task:  arn:aws:ecs:... TaskStatus: (NONE->RUNNING) Containers:  (CREATED->RUNNING),~internal~ecs~pause (RESOURCES_PROVISIONED->RESOURCES_PROVISIONED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]
00:12Z [INFO] Managed task arn:... container change also resulted in task change  [RUNNING]
00:12Z [INFO] Managed task arn:... unable to create task state change event []: create task state change event api: status not recognized by ECS: CREATED
00:12Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:12Z [INFO] Managed task arn:... waiting for event for task
00:12Z [INFO] Task engine arn:... starting container: 
00:12Z [INFO] Managed task arn:... got container  event: [RUNNING]
00:12Z [INFO] Managed task arn:... handling container change [{RUNNING {... <nil> [] <nil> []   com.amazonaws.ecs.task-arn:arn:aws:ecs:...  com.amazonaws.ecs.task-definition-version:15]    {UNKNOWN <nil> 0 } container: 0xc0007aeb00} ContainerStatusChangeEvent}] for container 
00:12Z [INFO] Managed task arn:... sending container change event  arn:aws:ecs:...  -> RUNNING, Known Sent: NONE
00:12Z [INFO] Managed task arn:... sent container change event  arn:aws:ecs:...  -> RUNNING, Known Sent: NONE
00:12Z [INFO] api/task: Updating task's known status to: RUNNING, task:  arn:aws:ecs:... TaskStatus: (CREATED->RUNNING) Containers:  (RUNNING->RUNNING),~internal~ecs~pause (RESOURCES_PROVISIONED->RESOURCES_PROVISIONED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]
00:12Z [INFO] Managed task arn:... container change also resulted in task change  [RUNNING]
00:12Z [INFO] Managed task arn:... sending task change event arn:... -> RUNNING, Known Sent: NONE, PullStartedAt:   PullStoppedAt:   ExecutionStoppedAt: ]
00:12Z [INFO] TaskHandler: batching container event  -> RUNNING, Known Sent: NONE
00:12Z [INFO] TaskHandler: Adding event: TaskChange: arn:... -> RUNNING, Known Sent: NONE, PullStartedAt:   PullStoppedAt:   ExecutionStoppedAt: , arn:aws:ecs:...  -> RUNNING, Known Sent: NONE] sent: false
00:12Z [INFO] TaskHandler: Sending task change: TaskChange: arn:... -> RUNNING, Known Sent: NONE, PullStartedAt:   PullStoppedAt:   ExecutionStoppedAt: , arn:aws:ecs:...  -> RUNNING, Known Sent: NONE] sent: false
00:12Z [INFO] Managed task arn:... sent task change event arn:... -> RUNNING, Known Sent: NONE, PullStartedAt:   PullStoppedAt:   ExecutionStoppedAt: ]
00:12Z [INFO] Managed task arn:... task at steady state: RUNNING
00:12Z [INFO] Managed task arn:... waiting for event for task
00:12Z [INFO] Task engine arn:... started docker container for task:  -> ..., took 204.620728ms
00:12Z [INFO] Managed task arn:... got container  event: [RUNNING]
00:12Z [INFO] Managed task arn:... handling container change [{RUNNING {... <nil> [] <nil> []   com.amazonaws.ecs.task-arn:arn:aws:ecs:...  com.amazonaws.ecs.task-definition-version:15]    {UNKNOWN <nil> 0 } container: 0xc00062f600} ContainerStatusChangeEvent}] for container 
00:12Z [INFO] Managed task arn:... redundant container state change.  to RUNNING, but already RUNNING
00:12Z [INFO] Managed task arn:... task at steady state: RUNNING
00:12Z [INFO] Managed task arn:... waiting for event for task
00:14Z [INFO] Managed task arn:... got container  event: [STOPPED]
00:14Z [INFO] Managed task arn:... handling container change [{STOPPED {... 0xc00 {UNKNOWN <nil> 0 } container: 0xc00062f800} ContainerStatusChangeEvent}] for container 
00:14Z [INFO] Task arn:... recording execution stopped time. Essential container  stopped at:  
00:14Z [INFO] Managed task arn:... sending container change event  arn:aws:ecs:...  -> STOPPED, Exit 0, , Known Sent: RUNNING
00:14Z [INFO] Managed task arn:... sent container change event  arn:aws:ecs:...  -> STOPPED, Exit 0, , Known Sent: RUNNING
00:14Z [INFO] api/task: Updating task desired status to stopped because of container:  task:  arn:aws:ecs:... TaskStatus: (RUNNING->RUNNING) Containers:  (STOPPED->RUNNING),~internal~ecs~pause (RESOURCES_PROVISIONED->RESOURCES_PROVISIONED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [INFO] Managed task arn:... got resource [cgroup] event: [REMOVED]
00:14Z [INFO] Task engine arn:... stopping container [~internal~ecs~pause]
00:14Z [INFO] Task engine arn:... cleaning up the network namespace
00:14Z [INFO] TaskHandler: batching container event  -> STOPPED, Exit 0, , Known Sent: RUNNING
00:14Z [INFO] api/task: Updating task desired status to stopped because of container:  task:  arn:aws:ecs:... TaskStatus: (RUNNING->STOPPED) Containers:  (STOPPED->STOPPED),~internal~ecs~pause (RESOURCES_PROVISIONED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: [~internal~ecs~pause]; task:  arn:aws:ecs:... TaskStatus: (RUNNING->STOPPED) Containers:  (STOPPED->STOPPED),~internal~ecs~pause (RESOURCES_PROVISIONED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] Managed task arn:... task not steady state or terminal; progressing it
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [WARN] DockerGoClient: Unable to decode stats for container ...: context canceled
00:14Z [INFO] Container ... is terminal, stopping stats collection
00:14Z [INFO] Task engine arn:... cleaned pause container network namespace
00:14Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [STOPPED]
00:14Z [INFO] Managed task arn:... handling container change [{STOPPED {... 0xc00 {UNKNOWN <nil> 0 } none 0xc00062fb00} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:14Z [INFO] Managed task arn:... unable to create state change event for container [~internal~ecs~pause]: create container state change event api: internal container: ~internal~ecs~pause
00:14Z [INFO] api/task: Updating task's known status to: STOPPED, task:  arn:aws:ecs:... TaskStatus: (RUNNING->STOPPED) Containers:  (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]
00:14Z [INFO] api/task: Updating task desired status to stopped because of container:  task:  arn:aws:ecs:... TaskStatus: (STOPPED->STOPPED) Containers:  (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: [~internal~ecs~pause]; task:  arn:aws:ecs:... TaskStatus: (STOPPED->STOPPED) Containers:  (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] Managed task arn:... container change also resulted in task change [~internal~ecs~pause]: [STOPPED]
00:14Z [INFO] Managed task arn:... sending task change event arn:... -> STOPPED, Known Sent: RUNNING, PullStartedAt:   PullStoppedAt:   ExecutionStoppedAt:  
00:14Z [INFO] Managed task arn:... sent task change event arn:... -> STOPPED, Known Sent: RUNNING, PullStartedAt:   PullStoppedAt:   ExecutionStoppedAt:  
00:14Z [INFO] api/task: Updating task desired status to stopped because of container:  task:  arn:aws:ecs:... TaskStatus: (STOPPED->STOPPED) Containers:  (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] api/task: Updating task desired status to stopped because of container: [~internal~ecs~pause]; task:  arn:aws:ecs:... TaskStatus: (STOPPED->STOPPED) Containers:  (STOPPED->STOPPED),~internal~ecs~pause (STOPPED->STOPPED),] ENIs: [eni id:eni-1 mac: aaa hostname: hostname ipv4addresses: [1.2.3.4], ipv6addresses: [], dns: [], dns search: [], gateway ipv4: [1.2.3.4/20][ ,VLan ID: [], TrunkInterfaceMacAddress: []],]]
00:14Z [INFO] Managed task arn:... task has reached stopped. Waiting for container cleanup
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [WARN] DockerGoClient: Unable to decode stats for container ...: context canceled
00:14Z [INFO] Container ... is terminal, stopping stats collection
00:14Z [INFO] TaskHandler: Adding event: TaskChange: arn:... -> STOPPED, Known Sent: RUNNING, PullStartedAt:   PullStoppedAt:   ExecutionStoppedAt:   arn:aws:ecs:...  -> STOPPED, Exit 0, , Known Sent: RUNNING] sent: false
00:14Z [INFO] TaskHandler: Sending task change: TaskChange: arn:... -> STOPPED, Known Sent: RUNNING, PullStartedAt:   PullStoppedAt:   ExecutionStoppedAt:   arn:aws:ecs:...  -> STOPPED, Exit 0, , Known Sent: RUNNING] sent: false
00:14Z [INFO] Managed task arn:... IPAM releasing ip for task eni
00:14Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [STOPPED]
00:14Z [INFO] Managed task arn:... handling container change [{STOPPED {... 0xc000489ee0 [] <nil> []  com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:...  com.amazonaws.ecs.task-definition-version:15]   {UNKNOWN <nil> 0 } none 0xc000505000} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:14Z [INFO] Managed task arn:... redundant container state change. ~internal~ecs~pause to STOPPED, but already STOPPED
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [INFO] Managed task arn:... got container [~internal~ecs~pause] event: [STOPPED]
00:14Z [INFO] Managed task arn:... handling container change [{STOPPED {... 0xc000072500 [] <nil> []  com.amazonaws.ecs.container-name:~internal~ecs~pause com.amazonaws.ecs.task-arn:arn:aws:ecs:...  com.amazonaws.ecs.task-definition-version:15]   {UNKNOWN <nil> 0 } none 0xc000505400} ContainerStatusChangeEvent}] for container [~internal~ecs~pause]
00:14Z [INFO] Managed task arn:... redundant container state change. ~internal~ecs~pause to STOPPED, but already STOPPED
00:14Z [INFO] Managed task arn:... waiting for event for task
00:14Z [WARN] Udev watcher event-handler: unable to send state change: udev watcher send ENI state change: eni status already sent: ENI Attachment: 
00:52Z [INFO] Handling ENI attachment
00:52Z [INFO] Starting ENI ack timer 
00:52Z [INFO] Adding task eni attachment 

任何人都可以据此指出潜在原因吗? 还有其他帮助吗?

0 个答案:

没有答案