我遇到flink应用程序失败的问题。
此流式传输作业在部署纱线后不久就会运行。
但是在几分钟后因以下错误消息而失败。
它是低性能纱线组中高负荷的证据吗？
从flink开头
Log Type: jobmanager.log Log Upload Time: Tue Jun 12 18:19:50 +0900 2018 Log Length: 10807897 2018-06-11 18:59:27,167 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2018-06-11 18:59:27,168 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @ 14:54:44 UTC) 2018-06-11 18:59:27,168 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current user: irteam 2018-06-11 18:59:27,472 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-06-11 18:59:27,536 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current Hadoop/Kerberos user: irteam 2018-06-11 18:59:27,536 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14 2018-06-11 18:59:27,536 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap size: 66667 MiBytes 2018-06-11 18:59:27,537 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64 2018-06-11 18:59:27,537 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Hadoop version: 2.8.3 2018-06-11 18:59:27,537 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options: 2018-06-11 18:59:27,538 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx75000m 2018-06-11 18:59:27,538 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Djava.library.path=/home1/irteam/realtime-tools 2018-06-11 18:59:27,538 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog.file=/naver/search-cluster/eye/var/logs/application_1528711080009_0002/container_e08_1528711080009_0002_01_000001/jobmanager.log 2018-06-11 18:59:27,538 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlogback.configurationFile=file:logback.xml 2018-06-11 18:59:27,538 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Dlog4j.configuration=file:log4j.properties 2018-06-11 18:59:27,538 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program Arguments: (none) 2018-06-11 18:59:27,538 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Class path[omit] 2018-06-11 18:59:27,539 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -------------------------------------------------------------------------------- 2018-06-11 18:59:27,539 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX signal handlers for [TERM, HUP, INT] 2018-06-11 18:59:27,542 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - YARN daemon is running as: irteam Yarn client user obtainer: irteam 2018-06-11 18:59:27,544 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.home, "/usr/lib/jvm/java-1.8.0-openjdk" 2018-06-11 18:59:27,544 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.opts, "-Djava.library.path=/home1/irteam/realtime-tools" 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: high-availability.cluster-id, application_1528711080009_0002 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, 0.0.0.0 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 100000 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.network.request-backoff.max, 100000 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.taskmanager.env.JAVA_HOME, /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: rest.port, 8081 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: internal.cluster.execution-mode, NORMAL 2018-06-11 18:59:27,545 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 480 2018-06-11 18:59:27,546 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 10 2018-06-11 18:59:27,546 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 100000 2018-06-11 18:59:27,546 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: containerized.master.env.JAVA_HOME, /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64 2018-06-11 18:59:27,558 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Setting directories for temporary files to: /home1/irteam/naver/search-cluster/eye/volume/nodemanager/usercache/irteam/appcache/application_1528711080009_0002 2018-06-11 18:59:27,570 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting YarnSessionClusterEntrypoint. 2018-06-11 18:59:27,570 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default filesystem. 2018-06-11 18:59:27,636 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to irteam (auth:SIMPLE) 2018-06-11 18:59:27,650 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing cluster services. 2018-06-11 18:59:27,654 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start actor system at chd004.eye.nfra.io:33524 2018-06-11 18:59:28,126 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2018-06-11 18:59:28,222 INFO akka.remote.Remoting - Starting remoting 2018-06-11 18:59:28,322 INFO akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://flink@chd004.eye.nfra.io:33524] 2018-06-11 18:59:28,329 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@chd004.eye.nfra.io:33524 2018-06-11 18:59:28,348 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /home1/irteam/naver/search-cluster/eye/volume/nodemanager/usercache/irteam/appcache/application_1528711080009_0002/blobStore-c25d4d9d-4ddc-442d-8d5e-7bec36dca006 2018-06-11 18:59:28,349 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:45733 - max concurrent requests: 50 - max backlog: 1000 2018-06-11 18:59:28,363 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter configured, no metrics will be exposed/reported. 2018-06-11 18:59:28,367 INFO org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore - Initializing FileArchivedExecutionGraphStore: Storage directory /home1/irteam/naver/search-cluster/eye/volume/nodemanager/usercache/irteam/appcache/application_1528711080009_0002/executionGraphStore-63bcf196-410d-4d8c-8388-f270beb53555, expiration time 3600000, maximum cache size 52428800 bytes. 2018-06-11 18:59:28,388 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /home1/irteam/naver/search-cluster/eye/volume/nodemanager/usercache/irteam/appcache/application_1528711080009_0002/blobStore-02db740f-8c23-46e8-bb24-1f583b6a0b33 2018-06-11 18:59:28,395 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload directory /tmp/flink-web-8698d702-67fe-437c-b62e-78c2969bf770/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 2018-06-11 18:59:28,396 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created directory /tmp/flink-web-8698d702-67fe-437c-b62e-78c2969bf770/flink-web-upload for file uploads. 2018-06-11 18:59:28,399 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest endpoint. 2018-06-11 18:59:28,737 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component log file: /naver/search-cluster/eye/var/logs/application_1528711080009_0002/container_e08_1528711080009_0002_01_000001/jobmanager.log 2018-06-11 18:59:28,737 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of main cluster component stdout file: /naver/search-cluster/eye/var/logs/application_1528711080009_0002/container_e08_1528711080009_0002_01_000001/jobmanager.out 2018-06-11 18:59:28,808 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint listening at chd004.eye.nfra.io:39794 2018-06-11 18:59:28,808 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - http://chd004.eye.nfra.io:39794 was granted leadership with leaderSessionID=00000000-0000-0000-0000-000000000000 2018-06-11 18:59:28,808 INFO org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend listening at http://chd004.eye.nfra.io:39794. 2018-06-11 18:59:28,817 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.yarn.YarnResourceManager at akka://flink/user/resourcemanager . 2018-06-11 18:59:28,902 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.dispatcher.StandaloneDispatcher at akka://flink/user/dispatcher . 2018-06-11 18:59:28,916 INFO org.apache.flink.yarn.YarnResourceManager - ResourceManager akka.tcp://flink@chd004.eye.nfra.io:33524/user/resourcemanager was granted leadership with fencing token 00000000000000000000000000000000 2018-06-11 18:59:28,917 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting the SlotManager. 2018-06-11 18:59:29,161 INFO org.apache.flink.yarn.YarnResourceManager - Recovered 0 containers from previous attempts ([]). 2018-06-11 18:59:29,163 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-cached-nodemanagers-proxies : 0 2018-06-11 18:59:29,174 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher akka.tcp://flink@chd004.eye.nfra.io:33524/user/dispatcher was granted leadership with fencing token 00000000000000000000000000000000 2018-06-11 18:59:29,174 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all persisted jobs. 2018-06-11 18:59:31,120 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job 5f090c4f4287db062cee0996da5d5ffc (LCS realtime data). 2018-06-11 18:59:31,130 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 . 2018-06-11 18:59:31,136 INFO org.apache.flink.runtime.jobmaster.JobMaster - Initializing job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc). 2018-06-11 18:59:31,144 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=3, delayBetweenRestartAttempts=30000) for LCS realtime data (5f090c4f4287db062cee0996da5d5ffc). 2018-06-11 18:59:31,148 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting RPC endpoint for org.apache.flink.runtime.jobmaster.slotpool.SlotPool at akka://flink/user/a6ffe322-07db-4282-a29c-0836ad26cd9f . 2018-06-11 18:59:31,165 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2018-06-11 18:59:31,174 INFO org.apache.flink.runtime.jobmaster.JobMaster - Running initialization on master for job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc). 2018-06-11 18:59:31,174 INFO org.apache.flink.runtime.jobmaster.JobMaster - Successfully ran initialization on master in 0 ms. 2018-06-11 18:59:31,248 INFO org.apache.flink.runtime.jobmaster.JobMaster - Using application-defined state backend: File State Backend (checkpoints: 'file:/home1/irteam/apps/flink-1.4.0/checkpoint', savepoints: 'null', asynchronous: UNDEFINED, fileStateThreshold: -1) 2018-06-11 18:59:31,248 INFO org.apache.flink.runtime.jobmaster.JobMaster - Configuring application-defined state backend with job/cluster config 2018-06-11 18:59:31,258 INFO org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager runner for job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc) was granted leadership with session id 00000000-0000-0000-0000-000000000000 at akka.tcp://flink@chd004.eye.nfra.io:33524/user/jobmanager_0. 2018-06-11 18:59:31,260 INFO org.apache.flink.runtime.jobmaster.JobMaster - Starting execution of job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc) 2018-06-11 18:59:31,261 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc) switched from state CREATED to RUNNING. 2018-06-11 18:59:31,264 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source (1/480) (98a01166bb2ac99dd301e4b60febbc45) switched from CREATED to SCHEDULED.
在超时事件附近可能导致flink作业失败。
2018-06-12 18:17:39,750 INFO org.apache.flink.runtime.rest.handler.legacy.backpressure.StackTraceSampleCoordinator - Cancelling sample 5589
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink@chd023.eye.nfra.io:34783/user/taskmanager_0#-297572584]] after [15000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation".
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
at java.lang.Thread.run(Thread.java:748)
2018-06-12 18:17:39,770 INFO org.apache.flink.runtime.rest.handler.legacy.backpressure.StackTraceSampleCoordinator - Cancelling sample 5590
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink@chd032.eye.nfra.io:34653/user/taskmanager_0#424015125]] after [15000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation".
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
at java.lang.Thread.run(Thread.java:748)
2018-06-12 18:17:51,270 INFO org.apache.flink.runtime.rest.handler.legacy.backpressure.StackTraceSampleCoordinator - Cancelling sample 5591
akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink@chd032.eye.nfra.io:34653/user/taskmanager_0#424015125]] after [15000 ms]. Sender[null] sent message of type "org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation".
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
at java.lang.Thread.run(Thread.java:748)
2018-06-12 18:17:55,650 INFO org.apache.flink.yarn.YarnResourceManager - The heartbeat of TaskManager with id container_e08_1528711080009_0002_01_000017 timed out.
2018-06-12 18:17:55,650 INFO org.apache.flink.yarn.YarnResourceManager - Closing TaskExecutor connection container_e08_1528711080009_0002_01_000017 because: The heartbeat of TaskManager with id container_e08_1528711080009_0002_01_000017 timed out.
2018-06-12 18:17:55,650 INFO org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Unregister TaskManager 525095d833344e8b205017666accd9c5 from the SlotManager.
2018-06-12 18:17:55,650 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Window(EventTimeSessionWindows(300000), NowTrigger, NowSessionProcessor) -> Sink: Unnamed (188/480) (f9ed2fc23d6ca5a364300864b60760af) switched from RUNNING to FAILED.
org.apache.flink.util.FlinkException: Releasing TaskManager container_e08_1528711080009_0002_01_000017.
at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManagerInternal(SlotPool.java:1067)
at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManager(SlotPool.java:1050)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-06-12 18:17:55,651 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job LCS realtime data (5f090c4f4287db062cee0996da5d5ffc) switched from state RUNNING to FAILING.
org.apache.flink.util.FlinkException: Releasing TaskManager container_e08_1528711080009_0002_01_000017.
at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManagerInternal(SlotPool.java:1067)
at org.apache.flink.runtime.jobmaster.slotpool.SlotPool.releaseTaskManager(SlotPool.java:1050)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:247)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-06-12 18:17:55,679 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source (1/480) (98a01166bb2ac99dd301e4b60febbc45) switched from RUNNING to CANCELING.