我正在尝试在群集上使用yarn-client模式启动Spark作业。我已经尝试过带有纱线的spark-shell，我可以启动应用程序。但是，我也希望能够运行驱动程序，比如eclipse，同时使用集群来运行任务。我还为HDFS添加了spark-assembly jar并通过向eclipse添加（HADOOP_CONF_DIR env变量）来指向它，虽然我不确定这是否是最好的方法。
我的应用程序确实在群集上启动（正如我在资源管理器的监视器中看到的那样）它“成功”完成但没有任何结果返回给驱动程序。我在eclipse控制台中看到以下异常：
WARN 10:11:08,375 Logging.scala:71 -- Lost task 0.0 in stage 1.0 (TID 1, vanpghdcn2.pgdev.sap.corp): java.lang.NullPointerException
at org.apache.spark.sql.execution.RDDConversions$$anonfun$rowToRowRdd$1.apply(ExistingRDD.scala:56)
at org.apache.spark.sql.execution.RDDConversions$$anonfun$rowToRowRdd$1.apply(ExistingRDD.scala:55)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:686)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$17.apply(RDD.scala:686)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
ERROR 10:11:08,522 Logging.scala:75 -- Task 0 in stage 1.0 failed 4 times; aborting job
INFO 10:11:08,538 SparkUtils.scala:67 -- SparkContext stopped
我在ResourceManager日志中得到以下内容：
2015-10-22 10:10:42,603 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1445462013958_0012_000001 container=Container: [ContainerId: container_1445462013958_0012_01_000001, NodeId: vanpghdcn3.pgdev.sap.corp:41419, NodeHttpAddress: vanpghdcn3.pgdev.sap.corp:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 clusterResource=<memory:24576, vCores:24>
2015-10-22 10:10:42,603 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.041666668, absoluteUsedCapacity=0.041666668, numApps=1, numContainers=1
2015-10-22 10:10:42,603 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.041666668 absoluteUsedCapacity=0.041666668 used=<memory:1024, vCores:1> cluster=<memory:24576, vCores:24>
2015-10-22 10:10:42,604 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : vanpghdcn3.pgdev.sap.corp:41419 for container : container_1445462013958_0012_01_000001
2015-10-22 10:10:42,606 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000001 Container Transitioned from ALLOCATED to ACQUIRED
2015-10-22 10:10:42,606 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Clear node set for appattempt_1445462013958_0012_000001
2015-10-22 10:10:42,606 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1445462013958_0012 AttemptId: appattempt_1445462013958_0012_000001 MasterContainer: Container: [ContainerId: container_1445462013958_0012_01_000001, NodeId: vanpghdcn3.pgdev.sap.corp:41419, NodeHttpAddress: vanpghdcn3.pgdev.sap.corp:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.165.28.145:41419 }, ]
2015-10-22 10:10:42,606 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1445462013958_0012_000001 State change from SCHEDULED to ALLOCATED_SAVING
2015-10-22 10:10:42,606 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1445462013958_0012_000001 State change from ALLOCATED_SAVING to ALLOCATED
2015-10-22 10:10:42,606 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1445462013958_0012_000001
2015-10-22 10:10:42,608 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1445462013958_0012_01_000001, NodeId: vanpghdcn3.pgdev.sap.corp:41419, NodeHttpAddress: vanpghdcn3.pgdev.sap.corp:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.165.28.145:41419 }, ] for AM appattempt_1445462013958_0012_000001
2015-10-22 10:10:42,608 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1445462013958_0012_01_000001 : {{JAVA_HOME}}/bin/java,-server,-Xmx512m,-Djava.io.tmpdir={{PWD}}/tmp,'-Dspark.eventLog.dir=','-Dspark.driver.port=57819','-Dspark.app.name=Sparca Application','-Dspark.executor.memory=1g','-Dspark.master=yarn-client','-Dspark.executor.id=driver','-Dspark.externalBlockStore.folderName=spark-10391661-8d35-40d9-8242-fe79bdc19d2d','-Dspark.fileserver.uri=<a href="http://10.161.43.118:57820','-Dspark.driver.appUIAddress=http://10.161.43.118:4040','-Dspark.driver.host=10.161.43.118','-Dspark.eventLog.enabled=false','-Dspark.yarn.jar=hdfs://vanpghdcn1.pgdev.sap.corp:8020/data/spark-assembly-1.4.0-hadoop2.6.0.jar','-Dspark.cores.max=6',-Dspark.yarn.app.container.log.dir=">http://10.161.43.118:57820','-Dspark.driver.appUIAddress=http://10.161.43.118:4040','-Dspark.driver.host=10.161.43.118','-Dspark.eventLog.enabled=false','-Dspark.yarn.jar=hdfs://vanpghdcn1.pgdev.sap.corp:8020/data/spark-assembly-1.4.0-hadoop2.6.0.jar','-Dspark.cores.max=6',-Dspark.yarn.app.container.log.dir=<LOG_DIR>,org.apache.spark.deploy.yarn.ExecutorLauncher,--arg,'10.161.43.118:57819',--executor-memory,1024m,--executor-cores,1,--num-executors ,2,1>,<LOG_DIR>/stdout,2>,<LOG_DIR>/stderr
2015-10-22 10:10:42,608 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1445462013958_0012_000001
2015-10-22 10:10:42,608 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1445462013958_0012_000001
2015-10-22 10:10:42,640 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1445462013958_0012_01_000001, NodeId: vanpghdcn3.pgdev.sap.corp:41419, NodeHttpAddress: vanpghdcn3.pgdev.sap.corp:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.165.28.145:41419 }, ] for AM appattempt_1445462013958_0012_000001
2015-10-22 10:10:42,640 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1445462013958_0012_000001 State change from ALLOCATED to LAUNCHED
2015-10-22 10:10:43,613 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000001 Container Transitioned from ACQUIRED to RUNNING
2015-10-22 10:10:48,176 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1445462013958_0012_000001 (auth:SIMPLE)
2015-10-22 10:10:48,188 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AM registration appattempt_1445462013958_0012_000001
2015-10-22 10:10:48,188 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hdfs IP=10.165.28.145 OPERATION=Register App Master TARGET=ApplicationMasterService RESULT=SUCCESS APPID=application_1445462013958_0012 APPATTEMPTID=appattempt_1445462013958_0012_000001
2015-10-22 10:10:48,188 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1445462013958_0012_000001 State change from LAUNCHED to RUNNING
2015-10-22 10:10:48,188 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1445462013958_0012 State change from ACCEPTED to RUNNING
2015-10-22 10:10:48,632 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000002 Container Transitioned from NEW to ALLOCATED
2015-10-22 10:10:48,632 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hdfs OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1445462013958_0012 CONTAINERID=container_1445462013958_0012_01_000002
2015-10-22 10:10:48,632 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1445462013958_0012_01_000002 of capacity <memory:2048, vCores:1> on host vanpghdcn3.pgdev.sap.corp:41419, which has 2 containers, <memory:3072, vCores:2> used and <memory:5120, vCores:6> available after allocation
2015-10-22 10:10:48,632 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1445462013958_0012_000001 container=Container: [ContainerId: container_1445462013958_0012_01_000002, NodeId: vanpghdcn3.pgdev.sap.corp:41419, NodeHttpAddress: vanpghdcn3.pgdev.sap.corp:8042, Resource: <memory:2048, vCores:1>, Priority: 1, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.041666668, absoluteUsedCapacity=0.041666668, numApps=1, numContainers=1 clusterResource=<memory:24576, vCores:24>
2015-10-22 10:10:48,633 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:3072, vCores:2>, usedCapacity=0.125, absoluteUsedCapacity=0.125, numApps=1, numContainers=2
2015-10-22 10:10:48,633 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.125 absoluteUsedCapacity=0.125 used=<memory:3072, vCores:2> cluster=<memory:24576, vCores:24>
2015-10-22 10:10:48,819 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000003 Container Transitioned from NEW to ALLOCATED
2015-10-22 10:10:48,819 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hdfs OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1445462013958_0012 CONTAINERID=container_1445462013958_0012_01_000003
2015-10-22 10:10:48,819 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1445462013958_0012_01_000003 of capacity <memory:2048, vCores:1> on host vanpghdcn2.pgdev.sap.corp:36064, which has 1 containers, <memory:2048, vCores:1> used and <memory:6144, vCores:7> available after allocation
2015-10-22 10:10:48,819 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: assignedContainer application attempt=appattempt_1445462013958_0012_000001 container=Container: [ContainerId: container_1445462013958_0012_01_000003, NodeId: vanpghdcn2.pgdev.sap.corp:36064, NodeHttpAddress: vanpghdcn2.pgdev.sap.corp:8042, Resource: <memory:2048, vCores:1>, Priority: 1, Token: null, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:3072, vCores:2>, usedCapacity=0.125, absoluteUsedCapacity=0.125, numApps=1, numContainers=2 clusterResource=<memory:24576, vCores:24>
2015-10-22 10:10:48,819 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting assigned queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:5120, vCores:3>, usedCapacity=0.20833333, absoluteUsedCapacity=0.20833333, numApps=1, numContainers=3
2015-10-22 10:10:48,820 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: assignedContainer queue=root usedCapacity=0.20833333 absoluteUsedCapacity=0.20833333 used=<memory:5120, vCores:3> cluster=<memory:24576, vCores:24>
2015-10-22 10:10:53,253 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : vanpghdcn3.pgdev.sap.corp:41419 for container : container_1445462013958_0012_01_000002
2015-10-22 10:10:53,255 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000002 Container Transitioned from ALLOCATED to ACQUIRED
2015-10-22 10:10:53,256 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : vanpghdcn2.pgdev.sap.corp:36064 for container : container_1445462013958_0012_01_000003
2015-10-22 10:10:53,257 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000003 Container Transitioned from ALLOCATED to ACQUIRED
2015-10-22 10:10:53,643 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000002 Container Transitioned from ACQUIRED to RUNNING
2015-10-22 10:10:53,830 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000003 Container Transitioned from ACQUIRED to RUNNING
2015-10-22 10:10:58,282 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: checking for deactivate of application :application_1445462013958_0012
2015-10-22 10:11:08,349 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1445462013958_0012_000001 with final state: FINISHING, and exit status: -1000
2015-10-22 10:11:08,349 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1445462013958_0012_000001 State change from RUNNING to FINAL_SAVING
2015-10-22 10:11:08,349 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1445462013958_0012 with final state: FINISHING
2015-10-22 10:11:08,349 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1445462013958_0012 State change from RUNNING to FINAL_SAVING
2015-10-22 10:11:08,350 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1445462013958_0012
2015-10-22 10:11:08,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1445462013958_0012_000001 State change from FINAL_SAVING to FINISHING
2015-10-22 10:11:08,350 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1445462013958_0012 State change from FINAL_SAVING to FINISHING
2015-10-22 10:11:08,453 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: application_1445462013958_0012 unregistered successfully.
2015-10-22 10:11:08,692 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000002 Container Transitioned from RUNNING to COMPLETED
2015-10-22 10:11:08,692 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1445462013958_0012_01_000002 in state: COMPLETED event:FINISHED
2015-10-22 10:11:08,692 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hdfs OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1445462013958_0012 CONTAINERID=container_1445462013958_0012_01_000002
2015-10-22 10:11:08,693 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1445462013958_0012_01_000002 of capacity <memory:2048, vCores:1> on host vanpghdcn3.pgdev.sap.corp:41419, which currently has 1 containers, <memory:1024, vCores:1> used and <memory:7168, vCores:7> available, release resources=true
2015-10-22 10:11:08,693 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:3072, vCores:2> numContainers=2 user=hdfs user-resources=<memory:3072, vCores:2>
2015-10-22 10:11:08,693 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1445462013958_0012_01_000002, NodeId: vanpghdcn3.pgdev.sap.corp:41419, NodeHttpAddress: vanpghdcn3.pgdev.sap.corp:8042, Resource: <memory:2048, vCores:1>, Priority: 1, Token: Token { kind: ContainerToken, service: 10.165.28.145:41419 }, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:3072, vCores:2>, usedCapacity=0.125, absoluteUsedCapacity=0.125, numApps=1, numContainers=2 cluster=<memory:24576, vCores:24>
2015-10-22 10:11:08,693 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.125 absoluteUsedCapacity=0.125 used=<memory:3072, vCores:2> cluster=<memory:24576, vCores:24>
2015-10-22 10:11:08,693 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:3072, vCores:2>, usedCapacity=0.125, absoluteUsedCapacity=0.125, numApps=1, numContainers=2
2015-10-22 10:11:08,693 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1445462013958_0012_000001 released container container_1445462013958_0012_01_000002 on node: host: vanpghdcn3.pgdev.sap.corp:41419 #containers=1 available=<memory:7168, vCores:7> used=<memory:1024, vCores:1> with event: FINISHED
2015-10-22 10:11:08,704 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8050: readAndProcess from client 10.161.43.118 threw exception [java.io.IOException: Connection reset by peer]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.apache.hadoop.ipc.Server.channelRead(Server.java:2603)
at org.apache.hadoop.ipc.Server.access$2800(Server.java:136)
at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1481)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:771)
at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:637)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:608)
2015-10-22 10:11:08,920 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000003 Container Transitioned from RUNNING to COMPLETED
2015-10-22 10:11:08,920 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1445462013958_0012_01_000003 in state: COMPLETED event:FINISHED
2015-10-22 10:11:08,920 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hdfs OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1445462013958_0012 CONTAINERID=container_1445462013958_0012_01_000003
2015-10-22 10:11:08,920 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1445462013958_0012_01_000003 of capacity <memory:2048, vCores:1> on host vanpghdcn2.pgdev.sap.corp:36064, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2015-10-22 10:11:08,920 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:1024, vCores:1> numContainers=1 user=hdfs user-resources=<memory:1024, vCores:1>
2015-10-22 10:11:08,921 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1445462013958_0012_01_000003, NodeId: vanpghdcn2.pgdev.sap.corp:36064, NodeHttpAddress: vanpghdcn2.pgdev.sap.corp:8042, Resource: <memory:2048, vCores:1>, Priority: 1, Token: Token { kind: ContainerToken, service: 10.165.28.143:36064 }, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.041666668, absoluteUsedCapacity=0.041666668, numApps=1, numContainers=1 cluster=<memory:24576, vCores:24>
2015-10-22 10:11:08,921 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.041666668 absoluteUsedCapacity=0.041666668 used=<memory:1024, vCores:1> cluster=<memory:24576, vCores:24>
2015-10-22 10:11:08,921 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:1024, vCores:1>, usedCapacity=0.041666668, absoluteUsedCapacity=0.041666668, numApps=1, numContainers=1
2015-10-22 10:11:08,921 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1445462013958_0012_000001 released container container_1445462013958_0012_01_000003 on node: host: vanpghdcn2.pgdev.sap.corp:36064 #containers=0 available=<memory:8192, vCores:8> used=<memory:0, vCores:0> with event: FINISHED
2015-10-22 10:11:09,694 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1445462013958_0012_01_000001 Container Transitioned from RUNNING to COMPLETED
2015-10-22 10:11:09,694 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1445462013958_0012_000001
2015-10-22 10:11:09,694 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: Completed container: container_1445462013958_0012_01_000001 in state: COMPLETED event:FINISHED
2015-10-22 10:11:09,694 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hdfs OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1445462013958_0012 CONTAINERID=container_1445462013958_0012_01_000001
2015-10-22 10:11:09,695 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1445462013958_0012_01_000001 of capacity <memory:1024, vCores:1> on host vanpghdcn3.pgdev.sap.corp:41419, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2015-10-22 10:11:09,695 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: default used=<memory:0, vCores:0> numContainers=0 user=hdfs user-resources=<memory:0, vCores:0>
2015-10-22 10:11:09,695 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: completedContainer container=Container: [ContainerId: container_1445462013958_0012_01_000001, NodeId: vanpghdcn3.pgdev.sap.corp:41419, NodeHttpAddress: vanpghdcn3.pgdev.sap.corp:8042, Resource: <memory:1024, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 10.165.28.145:41419 }, ] queue=default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0 cluster=<memory:24576, vCores:24>
2015-10-22 10:11:09,695 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: completedContainer queue=root usedCapacity=0.0 absoluteUsedCapacity=0.0 used=<memory:0, vCores:0> cluster=<memory:24576, vCores:24>
2015-10-22 10:11:09,695 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Re-sorting completed queue: root.default stats: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>, usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0
2015-10-22 10:11:09,695 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application attempt appattempt_1445462013958_0012_000001 released container container_1445462013958_0012_01_000001 on node: host: vanpghdcn3.pgdev.sap.corp:41419 #containers=0 available=<memory:8192, vCores:8> used=<memory:0, vCores:0> with event: FINISHED
2015-10-22 10:11:09,694 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1445462013958_0012_000001
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1445462013958_0012_000001 State change from FINISHING to FINISHED
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1445462013958_0012 State change from FINISHING to FINISHED
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1445462013958_0012_000001 is done. finalState=FINISHED
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hdfs OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1445462013958_0012
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1445462013958_0012 requests cleared
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1445462013958_0012 user: hdfs queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1445462013958_0012 user: hdfs leaf-queue of parent: root #applications: 0
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1445462013958_0012,name=Sparca Application,user=hdfs,queue=default,state=FINISHED,trackingUrl=http://vanpghdcn1:8088/proxy/application_1445462013958_0012/,appMasterHost=10.165.28.145,startTime=1445533842429,finishTime=1445533868349,finalStatus=SUCCEEDED,memorySeconds=109990,vcoreSeconds=67,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0>,applicationType=SPARK
2015-10-22 10:11:09,696 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1445462013958_0012_000001
2015-10-22 10:11:10,719 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Null container completed...
2015-10-22 10:11:10,925 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Null container completed...
值得一提的是，10.161.43.118是我正在运行我的日食的机器。我的测试应用只是将csv读入数据帧并进行计数。
答案 0 :(得分：1)
您的文件大小有多大？ 这说，某种客户端网络问题暂时导致主机和端点之间的通信问题。 如果文件大小足够大，那么通过TCP通道传输数据需要很长时间，并且连接不能长时间处于活动状态并被重置。希望，这个答案有所帮助。
答案 1 :(得分：0)
我的猜测是两者之间存在一些防火墙问题。我首先要验证您的群集机器和设备。能够与司机主人交谈。