以下命令适用于客户端模式。我得到了OOM异常（我可能没有很好地利用配置和内存明智的属性）。我的应用程序中有一些操作可以通过ClusterManager在集群模式下更好地管理，但是我在集群模式下提交应用程序时会遇到以下异常。
在发布此问题之前，我提到了潜在的类似问题here，here和here，但我无法解决此问题。
我收到以下异常。
Exception in thread "main" org.apache.spark.SparkException: Application application_1474309493751_0045 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1092)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1139)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
异常前的日志：
16/10/17 19:58:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/17 19:58:54 INFO RMProxy: Connecting to ResourceManager at <cluster-node-2>/10.10.1.2:8050
16/10/17 19:58:55 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
16/10/17 19:58:55 INFO Client: Requesting a new application from cluster with 11 NodeManagers
16/10/17 19:58:55 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (5120 MB per container)
16/10/17 19:58:55 INFO Client: Will allocate AM container, with 1724 MB memory including 700 MB overhead
16/10/17 19:58:55 INFO Client: Setting up container launch context for our AM
16/10/17 19:58:55 INFO Client: Setting up the launch environment for our AM container
16/10/17 19:58:55 INFO Client: Using the spark assembly jar on HDFS because you are using HDP, defaultSparkAssembly:hdfs://<cluster-url>:8020/hdp/apps/2.4.2.0-258/spark/spark-hdp-assembly.jar
16/10/17 19:58:55 INFO Client: Preparing resources for our AM container
16/10/17 19:58:55 INFO Client: Using the spark assembly jar on HDFS because you are using HDP, defaultSparkAssembly:hdfs://<cluster-url>:8020/hdp/apps/2.4.2.0-258/spark/spark-hdp-assembly.jar
16/10/17 19:58:55 INFO Client: Source and destination file systems are the same. Not copying hdfs://<cluster-url>:8020/hdp/apps/2.4.2.0-258/spark/spark-hdp-assembly.jar
16/10/17 19:58:55 INFO Client: Uploading resource file:/home/asharma/application/scala/application-1.0-SNAPSHOT-jar-with-dependencies.jar -> hdfs://<cluster-url>:8020/user/asharma/.sparkStaging/application_1474309493751_0045/application-1.0-SNAPSHOT-jar-with-dependencies.jar
16/10/17 19:58:56 INFO Client: Uploading resource file:/usr/hdp/2.4.2.0-258/spark/lib/datanucleus-core-3.2.10.jar -> hdfs://<cluster-url>:8020/user/asharma/.sparkStaging/application_1474309493751_0045/datanucleus-core-3.2.10.jar
16/10/17 19:58:56 INFO Client: Uploading resource file:/usr/hdp/2.4.2.0-258/spark/lib/datanucleus-api-jdo-3.2.6.jar -> hdfs://<cluster-url>:8020/user/asharma/.sparkStaging/application_1474309493751_0045/datanucleus-api-jdo-3.2.6.jar
16/10/17 19:58:56 INFO Client: Uploading resource file:/usr/hdp/2.4.2.0-258/spark/lib/datanucleus-rdbms-3.2.9.jar -> hdfs://<cluster-url>:8020/user/asharma/.sparkStaging/application_1474309493751_0045/datanucleus-rdbms-3.2.9.jar
16/10/17 19:58:56 INFO Client: Uploading resource file:/etc/hive/conf/hive-site.xml -> hdfs://<cluster-url>:8020/user/asharma/.sparkStaging/application_1474309493751_0045/hive-site.xml
16/10/17 19:58:56 INFO Client: Uploading resource file:/tmp/spark-f9c532c0-4412-472a-a554-b0ae42473fc5/__spark_conf__7591539568905481709.zip -> hdfs://<cluster-url>:8020/user/asharma/.sparkStaging/application_1474309493751_0045/__spark_conf__7591539568905481709.zip
16/10/17 19:58:56 INFO SecurityManager: Changing view acls to: asharma
16/10/17 19:58:56 INFO SecurityManager: Changing modify acls to: asharma
16/10/17 19:58:56 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(asharma); users with modify permissions: Set(asharma)
16/10/17 19:58:56 INFO Client: Submitting application 45 to ResourceManager
16/10/17 19:58:56 INFO YarnClientImpl: Submitted application application_1474309493751_0045
16/10/17 19:58:57 INFO Client: Application report for application_1474309493751_0045 (state: ACCEPTED)
使用的提交命令如下：
spark-submit --class <driver-class>
--master yarn
--deploy-mode cluster
--jars /usr/hdp/2.4.2.0-258/spark/lib/datanucleus-core-3.2.10.jar,/usr/hdp/2.4.2.0-258/spark/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/2.4.2.0-258/spark/lib/datanucleus-rdbms-3.2.9.jar
--files /etc/hive/conf/hive-site.xml
<application-jar> <application-arguments>
修改
每个节点规范： 核心（CPU）：24 磁盘：~8TB 内存：125.81GB
为应用添加了YARN日志：
Container: container_e03_1474309493751_0050_01_000002 on machine02_45454
===========================================================================
LogType:directory.info
Log Upload Time:Wed Oct 19 15:51:14 -0600 2016
LogLength:7059
Log Contents:
ls -l:
total 20
lrwxrwxrwx 1 yarn hadoop 100 Oct 19 15:49 __app__.jar -> /hdfs/hadoop/yarn/local/usercache/USER/filecache/90/alamos-1.0-SNAPSHOT-jar-with-dependencies.jar
-rw-r--r-- 1 yarn hadoop 88 Oct 19 15:49 container_tokens
lrwxrwxrwx 1 yarn hadoop 84 Oct 19 15:49 datanucleus-api-jdo-3.2.6.jar -> /hdfs/hadoop/yarn/local/usercache/USER/filecache/91/datanucleus-api-jdo-3.2.6.jar
lrwxrwxrwx 1 yarn hadoop 82 Oct 19 15:49 datanucleus-core-3.2.10.jar -> /hdfs/hadoop/yarn/local/usercache/USER/filecache/92/datanucleus-core-3.2.10.jar
lrwxrwxrwx 1 yarn hadoop 82 Oct 19 15:49 datanucleus-rdbms-3.2.9.jar -> /hdfs/hadoop/yarn/local/usercache/USER/filecache/89/datanucleus-rdbms-3.2.9.jar
-rwx------ 1 yarn hadoop 676 Oct 19 15:49 default_container_executor_session.sh
-rwx------ 1 yarn hadoop 730 Oct 19 15:49 default_container_executor.sh
lrwxrwxrwx 1 yarn hadoop 68 Oct 19 15:49 hive-site.xml -> /hdfs/hadoop/yarn/local/usercache/USER/filecache/93/hive-site.xml
-rwx------ 1 yarn hadoop 7660 Oct 19 15:49 launch_container.sh
lrwxrwxrwx 1 yarn hadoop 92 Oct 19 15:49 __spark_conf__ -> /hdfs/hadoop/yarn/local/usercache/USER/filecache/88/__spark_conf__4459058499745956980.zip
lrwxrwxrwx 1 yarn hadoop 59 Oct 19 15:49 __spark__.jar -> /hdfs/hadoop/yarn/local/filecache/10/spark-hdp-assembly.jar
drwx--x--- 2 yarn hadoop 6 Oct 19 15:49 tmp
find -L . -maxdepth 5 -ls:
886578 4 drwx--x--- 3 yarn hadoop 4096 Oct 19 15:49 .
8599305697 0 drwx--x--- 2 yarn hadoop 6 Oct 19 15:49 ./tmp
886579 4 -rw-r--r-- 1 yarn hadoop 88 Oct 19 15:49 ./container_tokens
886580 4 -rw-r--r-- 1 yarn hadoop 12 Oct 19 15:49 ./.container_tokens.crc
886581 8 -rwx------ 1 yarn hadoop 7660 Oct 19 15:49 ./launch_container.sh
886582 4 -rw-r--r-- 1 yarn hadoop 68 Oct 19 15:49 ./.launch_container.sh.crc
886583 4 -rwx------ 1 yarn hadoop 676 Oct 19 15:49 ./default_container_executor_session.sh
886584 4 -rw-r--r-- 1 yarn hadoop 16 Oct 19 15:49 ./.default_container_executor_session.sh.crc
886585 4 -rwx------ 1 yarn hadoop 730 Oct 19 15:49 ./default_container_executor.sh
886586 4 -rw-r--r-- 1 yarn hadoop 16 Oct 19 15:49 ./.default_container_executor.sh.crc
886576 1848 -r-x------ 1 yarn hadoop 1890075 Oct 19 15:49 ./datanucleus-core-3.2.10.jar
17182041689 1768 -r-x------ 1 yarn hadoop 1809447 Oct 19 15:49 ./datanucleus-rdbms-3.2.9.jar
8599305695 20 -r-x------ 1 yarn hadoop 19515 Oct 19 15:49 ./hive-site.xml
8598739830 4 drwx------ 2 yarn hadoop 4096 Oct 19 15:49 ./__spark_conf__
8598739831 8 -r-x------ 1 yarn hadoop 6537 Oct 19 15:49 ./__spark_conf__/mapred-site.xml
8598739832 8 -r-x------ 1 yarn hadoop 5408 Oct 19 15:49 ./__spark_conf__/hadoop-env.sh
8598739833 4 -r-x------ 1 yarn hadoop 620 Oct 19 15:49 ./__spark_conf__/log4j.properties
8598739834 4 -r-x------ 1 yarn hadoop 2370 Oct 19 15:49 ./__spark_conf__/hadoop-metrics2.properties
8598739835 16 -r-x------ 1 yarn hadoop 15856 Oct 19 15:49 ./__spark_conf__/yarn-site.xml
8598739836 4 -r-x------ 1 yarn hadoop 3979 Oct 19 15:49 ./__spark_conf__/hadoop-env.cmd
8598739837 4 -r-x------ 1 yarn hadoop 1 Oct 19 15:49 ./__spark_conf__/dfs.exclude
8598739838 0 -r-x------ 1 yarn hadoop 0 Oct 19 15:49 ./__spark_conf__/yarn.exclude
8598739839 4 -r-x------ 1 yarn hadoop 3306 Oct 19 15:49 ./__spark_conf__/core-site.xml
8599305664 4 -r-x------ 1 yarn hadoop 1631 Oct 19 15:49 ./__spark_conf__/kms-log4j.properties
8599305665 4 -r-x------ 1 yarn hadoop 2250 Oct 19 15:49 ./__spark_conf__/yarn-env.cmd
8599305666 4 -r-x------ 1 yarn hadoop 884 Oct 19 15:49 ./__spark_conf__/ssl-client.xml
8599305668 4 -r-x------ 1 yarn hadoop 2035 Oct 19 15:49 ./__spark_conf__/capacity-scheduler.xml
8599305669 4 -r-x------ 1 yarn hadoop 3518 Oct 19 15:49 ./__spark_conf__/kms-acls.xml
8599305670 4 -r-x------ 1 yarn hadoop 2358 Oct 19 15:49 ./__spark_conf__/topology_script.py
8599305671 4 -r-x------ 1 yarn hadoop 758 Oct 19 15:49 ./__spark_conf__/mapred-site.xml.template
8599305672 4 -r-x------ 1 yarn hadoop 1335 Oct 19 15:49 ./__spark_conf__/configuration.xsl
8599305673 8 -r-x------ 1 yarn hadoop 5112 Oct 19 15:49 ./__spark_conf__/yarn-env.sh
8599305674 8 -r-x------ 1 yarn hadoop 6610 Oct 19 15:49 ./__spark_conf__/hdfs-site.xml
8599305675 4 -r-x------ 1 yarn hadoop 1020 Oct 19 15:49 ./__spark_conf__/commons-logging.properties
8599305676 4 -r-x------ 1 yarn hadoop 1029 Oct 19 15:49 ./__spark_conf__/container-executor.cfg
8599305677 8 -r-x------ 1 yarn hadoop 4221 Oct 19 15:49 ./__spark_conf__/task-log4j.properties
8599305678 4 -r-x------ 1 yarn hadoop 2490 Oct 19 15:49 ./__spark_conf__/hadoop-metrics.properties
8599305679 4 -r-x------ 1 yarn hadoop 856 Oct 19 15:49 ./__spark_conf__/mapred-env.sh
8599305680 4 -r-x------ 1 yarn hadoop 1602 Oct 19 15:49 ./__spark_conf__/health_check
8599305681 4 -r-x------ 1 yarn hadoop 2316 Oct 19 15:49 ./__spark_conf__/ssl-client.xml.example
8599305682 4 -r-x------ 1 yarn hadoop 1527 Oct 19 15:49 ./__spark_conf__/kms-env.sh
8599305683 4 -r-x------ 1 yarn hadoop 1308 Oct 19 15:49 ./__spark_conf__/hadoop-policy.xml
8599305684 4 -r-x------ 1 yarn hadoop 285 Oct 19 15:49 ./__spark_conf__/slaves
8599305685 4 -r-x------ 1 yarn hadoop 724 Oct 19 15:49 ./__spark_conf__/topology_mappings.data
8599305686 8 -r-x------ 1 yarn hadoop 4956 Oct 19 15:49 ./__spark_conf__/metrics.properties
8599305687 4 -r-x------ 1 yarn hadoop 1000 Oct 19 15:49 ./__spark_conf__/ssl-server.xml
8599305688 4 -r-x------ 1 yarn hadoop 951 Oct 19 15:49 ./__spark_conf__/mapred-env.cmd
8599305689 4 -r-x------ 1 yarn hadoop 2268 Oct 19 15:49 ./__spark_conf__/ssl-server.xml.example
8599305690 4 -r-x------ 1 yarn hadoop 945 Oct 19 15:49 ./__spark_conf__/taskcontroller.cfg
8599305691 8 -r-x------ 1 yarn hadoop 5511 Oct 19 15:49 ./__spark_conf__/kms-site.xml
8599305692 8 -r-x------ 1 yarn hadoop 4113 Oct 19 15:49 ./__spark_conf__/mapred-queues.xml.template
8599305693 4 -r-x------ 1 yarn hadoop 979 Oct 19 15:49 ./__spark_conf__/__spark_conf__.properties
238 181616 -rwxr-xr-x 1 yarn hadoop 185971201 Jul 26 04:15 ./__spark__.jar
4295707318 139492 -r-x------ 1 yarn hadoop 142838305 Oct 19 15:49 ./__app__.jar
12884944925 332 -r-x------ 1 yarn hadoop 339666 Oct 19 15:49 ./datanucleus-api-jdo-3.2.6.jar
broken symlinks(find -L . -maxdepth 5 -type l -ls):
End of LogType:directory.info
LogType:launch_container.sh
Log Upload Time:Wed Oct 19 15:51:14 -0600 2016
LogLength:7660
Log Contents:
#!/bin/bash
export SPARK_YARN_STAGING_DIR=".sparkStaging/application_1474309493751_0050"
export HADOOP_CONF_DIR="/usr/hdp/current/hadoop-client/conf"
export JAVA_HOME="/usr/jdk64/jdk1.8.0_60"
export SPARK_YARN_CACHE_FILES="hdfs://machine.networkdomain:8020/hdp/apps/2.4.2.0-258/spark/spark-hdp-assembly.jar#__spark__.jar,hdfs://machine.networkdomain:8020/user/USER/.sparkStaging/application_1474309493751_0050/alamos-1.0-SNAPSHOT-jar-with-dependencies.jar#__app__.jar,hdfs://machine.networkdomain:8020/user/USER/.sparkStaging/application_1474309493751_0050/datanucleus-core-3.2.10.jar#datanucleus-core-3.2.10.jar,hdfs://machine.networkdomain:8020/user/USER/.sparkStaging/application_1474309493751_0050/datanucleus-api-jdo-3.2.6.jar#datanucleus-api-jdo-3.2.6.jar,hdfs://machine.networkdomain:8020/user/USER/.sparkStaging/application_1474309493751_0050/datanucleus-rdbms-3.2.9.jar#datanucleus-rdbms-3.2.9.jar,hdfs://machine.networkdomain:8020/user/USER/.sparkStaging/application_1474309493751_0050/hive-site.xml#hive-site.xml"
export SPARK_LOG_URL_STDOUT="http://machine02:8042/node/containerlogs/container_e03_1474309493751_0050_01_000002/USER/stdout?start=-4096"
export NM_HOST="machine02"
export SPARK_YARN_CACHE_FILES_FILE_SIZES="185971201,142838305,1890075,339666,1809447,19515"
export SPARK_YARN_CACHE_ARCHIVES_TIME_STAMPS="1476913762392"
export LOGNAME="USER"
export JVM_PID="$$"
export PWD="/hdfs/hadoop/yarn/local/usercache/USER/appcache/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002"
export LOCAL_DIRS="/hdfs/hadoop/yarn/local/usercache/USER/appcache/application_1474309493751_0050"
export NM_HTTP_PORT="8042"
export LOG_DIRS="/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002"
export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
"
export NM_PORT="45454"
export SPARK_YARN_CACHE_FILES_TIME_STAMPS="1467302252238,1476913762241,1476913762283,1476913762305,1476913762331,1476913762348"
export USER="USER"
export HADOOP_YARN_HOME="/usr/hdp/current/hadoop-yarn-nodemanager"
export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.4.2.0-258/hadoop/lib/hadoop-lzo-0.6.0.2.4.2.0-258.jar:/etc/hadoop/conf/secure"
export SPARK_YARN_CACHE_ARCHIVES="hdfs://machine.networkdomain:8020/user/USER/.sparkStaging/application_1474309493751_0050/__spark_conf__4459058499745956980.zip#__spark_conf__"
export SPARK_YARN_CACHE_ARCHIVES_FILE_SIZES="107659"
export SPARK_YARN_MODE="true"
export SPARK_YARN_CACHE_FILES_VISIBILITIES="PUBLIC,PRIVATE,PRIVATE,PRIVATE,PRIVATE,PRIVATE"
export HADOOP_TOKEN_FILE_LOCATION="/hdfs/hadoop/yarn/local/usercache/USER/appcache/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/container_tokens"
export NM_AUX_SERVICE_spark_shuffle=""
export SPARK_USER="USER"
export SPARK_LOG_URL_STDERR="http://machine02:8042/node/containerlogs/container_e03_1474309493751_0050_01_000002/USER/stderr?start=-4096"
export SPARK_YARN_CACHE_ARCHIVES_VISIBILITIES="PRIVATE"
export HOME="/home/"
export CONTAINER_ID="container_e03_1474309493751_0050_01_000002"
export MALLOC_ARENA_MAX="4"
ln -sf "/hdfs/hadoop/yarn/local/usercache/USER/filecache/92/datanucleus-core-3.2.10.jar" "datanucleus-core-3.2.10.jar"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
ln -sf "/hdfs/hadoop/yarn/local/usercache/USER/filecache/89/datanucleus-rdbms-3.2.9.jar" "datanucleus-rdbms-3.2.9.jar"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
ln -sf "/hdfs/hadoop/yarn/local/usercache/USER/filecache/93/hive-site.xml" "hive-site.xml"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
ln -sf "/hdfs/hadoop/yarn/local/usercache/USER/filecache/88/__spark_conf__4459058499745956980.zip" "__spark_conf__"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
ln -sf "/hdfs/hadoop/yarn/local/filecache/10/spark-hdp-assembly.jar" "__spark__.jar"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
ln -sf "/hdfs/hadoop/yarn/local/usercache/USER/filecache/90/alamos-1.0-SNAPSHOT-jar-with-dependencies.jar" "__app__.jar"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
ln -sf "/hdfs/hadoop/yarn/local/usercache/USER/filecache/91/datanucleus-api-jdo-3.2.6.jar" "datanucleus-api-jdo-3.2.6.jar"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
# Creating copy of launch script
cp "launch_container.sh" "/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/launch_container.sh"
chmod 640 "/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/launch_container.sh"
# Determining directory contents
echo "ls -l:" 1>"/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/directory.info"
ls -l 1>>"/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/directory.info"
echo "find -L . -maxdepth 5 -ls:" 1>>"/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/directory.info"
find -L . -maxdepth 5 -ls 1>>"/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/directory.info"
echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/directory.info"
find -L . -maxdepth 5 -type l -ls 1>>"/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/directory.info"
exec /bin/bash -c "$JAVA_HOME/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms4320m -Xmx4320m -Djava.io.tmpdir=$PWD/tmp '-Dspark.driver.port=36343' '-Dspark.history.ui.port=18080' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002 org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@11.11.18:36343 --executor-id 1 --hostname machine02 --cores 1 --app-id application_1474309493751_0050 --user-class-path file:$PWD/__app__.jar --user-class-path file:$PWD/datanucleus-core-3.2.10.jar --user-class-path file:$PWD/datanucleus-api-jdo-3.2.6.jar --user-class-path file:$PWD/datanucleus-rdbms-3.2.9.jar 1> /hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/stdout 2> /hdfs/hadoop/yarn/log/application_1474309493751_0050/container_e03_1474309493751_0050_01_000002/stderr"
hadoop_shell_errorcode=$?
if [ $hadoop_shell_errorcode -ne 0 ]
then
exit $hadoop_shell_errorcode
fi
End of LogType:launch_container.sh
LogType:stderr
Log Upload Time:Wed Oct 19 15:51:14 -0600 2016
LogLength:2995
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hdfs/hadoop/yarn/local/filecache/10/spark-hdp-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
16/10/19 15:49:29 INFO CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]
16/10/19 15:49:29 INFO SecurityManager: Changing view acls to: yarn,USER
16/10/19 15:49:29 INFO SecurityManager: Changing modify acls to: yarn,USER
16/10/19 15:49:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, USER); users with modify permissions: Set(yarn, USER)
16/10/19 15:49:30 INFO SecurityManager: Changing view acls to: yarn,USER
16/10/19 15:49:30 INFO SecurityManager: Changing modify acls to: yarn,USER
16/10/19 15:49:30 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, USER); users with modify permissions: Set(yarn, USER)
16/10/19 15:49:30 INFO Slf4jLogger: Slf4jLogger started
16/10/19 15:49:30 INFO Remoting: Starting remoting
16/10/19 15:49:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkExecutorActorSystem@machine02:56997]
16/10/19 15:49:30 INFO Utils: Successfully started service 'sparkExecutorActorSystem' on port 56997.
16/10/19 15:49:30 INFO DiskBlockManager: Created local directory at /hdfs/hadoop/yarn/local/usercache/USER/appcache/application_1474309493751_0050/blockmgr-fbeea917-a8f4-4ed1-99fa-8f88d069777d
16/10/19 15:49:30 INFO MemoryStore: MemoryStore started with capacity 2.8 GB
16/10/19 15:49:31 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@11.11.18:36343
16/10/19 15:49:31 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
16/10/19 15:49:31 INFO Executor: Starting executor ID 1 on host machine02
16/10/19 15:49:31 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50532.
16/10/19 15:49:31 INFO NettyBlockTransferService: Server created on 50532
16/10/19 15:49:31 INFO BlockManagerMaster: Trying to register BlockManager
16/10/19 15:49:31 INFO BlockManagerMaster: Registered BlockManager
16/10/19 15:50:16 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
16/10/19 15:50:16 INFO MemoryStore: MemoryStore cleared
16/10/19 15:50:16 INFO BlockManager: BlockManager stopped
16/10/19 15:50:16 WARN CoarseGrainedExecutorBackend: An unknown (machine08:36343) driver disconnected.
16/10/19 15:50:16 ERROR CoarseGrainedExecutorBackend: Driver 11.11.18:36343 disassociated! Shutting down.
16/10/19 15:50:16 INFO ShutdownHookManager: Shutdown hook called
End of LogType:stderr
LogType:stdout
Log Upload Time:Wed Oct 19 15:51:14 -0600 2016
LogLength:0
Log Contents:
End of LogType:stdout
有人可以指导我使用有用的资源和配置吗？任何帮助将不胜感激！
感谢。