我正在尝试使用YARN作为Cluster Manager运行我的Spark Word Count程序。我正在使用Hadoop 2.6＆amp; YARN配置为以伪分布式模式运行。申请提交失败
spark-submit --master yarn --class sbook.helloworld.WordCount
target/scala-2.11/sparkbookapp_2.11-1.0.jar
src/main/resources/data.txt output
以下是我能够从节点管理器检索的错误（检查具有级别ERROR的日志）。看起来这似乎是容器分配的问题，但无法找出原因
INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1434487429379_0008_000002
INFO spark.SecurityManager: Changing view acls to: mountain
INFO spark.SecurityManager: Changing modify acls to: mountain
INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(mountain); users with modify permissions: Set(mountain)
WARN util.Utils: Your hostname, mountain resolves to a loopback address: 127.0.1.1; using 10.0.0.6 instead (on interface wlan0)
WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
INFO slf4j.Slf4jLogger: Slf4jLogger started
INFO Remoting: Starting remoting
INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkYarnAM@10.0.0.6:42711]
INFO util.Utils: Successfully started service 'sparkYarnAM' on port 42711.
INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
INFO yarn.ApplicationMaster: Driver now available: 10.0.0.6:34231
INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@10.0.0.6:34231/user/YarnScheduler
INFO yarn.ApplicationMaster: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> mountain, PROXY_URI_BASES -> http://mountain:8088/proxy/application_1434487429379_0008),/proxy/application_1434487429379_0008)
INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
15/06/16 16:11:02 INFO yarn.YarnRMClient: Registering the ApplicationMaster
INFO yarn.YarnAllocator: Will request 2 executor containers, each with 1 cores and 1408 MB memory including 384 MB overhead
INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>)
INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
INFO yarn.ApplicationMaster: Final app status: UNDEFINED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with UNDEFINED (diag message: Shutdown hook called before final status was reported.)
INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1434487429379_0008
我还尝试在spark-submit脚本中将执行程序的大小更改为1g。但这也没有帮助，节点管理器仍在尝试创建一个大小为1408 MB的执行程序。