我在emr提交了一份火花工作。它在客户端模式下运行良好。但在群集模式下运行时会显示以下error.in 代码我使用来自GitHub的sqs接收器 - imapi / spark-sqs-receiver。并尝试使用s3a从亚马逊s3访问。
16/10/21 07:10:00 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 71, ip-172-31-54-127.ec2.internal, partition 0,NODE_LOCAL, 2115 bytes)
16/10/21 07:10:00 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-172-31-54-127.ec2.internal:43789 (size: 2.4 KB, free: 511.5 MB)
16/10/21 07:10:01 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 72, ip-172-31-54-127.ec2.internal, partition 1,NODE_LOCAL, 2115 bytes)
16/10/21 07:10:01 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 71, ip-172-31-54-127.ec2.internal): java.lang.ExceptionInInitializerError
at sqs.sqstest.Streaming$1.call(Streaming.java:146)
at sqs.sqstest.Streaming$1.call(Streaming.java:97)
at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$fn$1$1.apply(JavaDStreamLike.scala:170)
at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$fn$1$1.apply(JavaDStreamLike.scala:170)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:189)
at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:401)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:59)
at sqs.sqstest.Streaming.<clinit>(Streaming.java:41)
... 15 more
16/10/21 07:10:01 INFO YarnClusterSchedulerBackend: Disabling executor 1.
16/10/21 07:10:01 INFO DAGScheduler: Executor lost: 1 (epoch 1)
16/10/21 07:10:01 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.
16/10/21 07:10:01 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, ip-172-31-54-127.ec2.internal, 43789)
16/10/21 07:10:01 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor
16/10/21 07:10:02 INFO YarnAllocator: Completed container container_1477032067457_0003_02_000002 on host: ip-172-31-54-127.ec2.internal (state: COMPLETE, exit status: 50)
16/10/21 07:10:02 WARN YarnAllocator: Container marked as failed: container_1477032067457_0003_02_000002 on host: ip-172-31-54-127.ec2.internal. Exit status: 50. Diagnostics: Exception from container-launch.
Container id: container_1477032067457_0003_02_000002
Exit code: 50
Stack trace: ExitCodeException exitCode=50:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 50
16/10/21 07:10:02 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1477032067457_0003_02_000002 on host: ip-172-31-54-127.ec2.internal. Exit status: 50. Diagnostics: Exception from container-launch.
Container id: container_1477032067457_0003_02_000002
Exit code: 50
Stack trace: ExitCodeException exitCode=50:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)