RandomForest通过Spark导致堆空间错误

时间：2016-12-22 09:37:17

标签： apache-spark-mllib hortonworks-data-platform

HDP-2.5.0.0使用Ambari 2.4.0.1，Spark 2.0.1。

我有一个Scala代码，它读取一个108MB的csv文件并使用RandomForest。

我运行以下命令：

/usr/hdp/current/spark2-client/bin/spark-submit --class samples.FuelModel --master yarn --deploy-mode cluster --driver-memory 8g spark-assembly-1.0.jar

List of jobs

由于OutOfMemory，当堆空间溢出时，至少有一个容器被杀死，控制台输出是：

16/12/22 09:07:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/12/22 09:07:25 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
16/12/22 09:07:26 INFO TimelineClientImpl: Timeline service address: http://l4326pp.sss.com:8188/ws/v1/timeline/
16/12/22 09:07:26 INFO AHSProxy: Connecting to Application History server at l4326pp.sss.com/138.106.33.132:10200
16/12/22 09:07:26 INFO Client: Requesting a new application from cluster with 4 NodeManagers
16/12/22 09:07:26 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (204800 MB per container)
16/12/22 09:07:26 INFO Client: Will allocate AM container, with 9011 MB memory including 819 MB overhead
16/12/22 09:07:26 INFO Client: Setting up container launch context for our AM
16/12/22 09:07:26 INFO Client: Setting up the launch environment for our AM container
16/12/22 09:07:26 INFO Client: Preparing resources for our AM container
16/12/22 09:07:27 INFO YarnSparkHadoopUtil: getting token for namenode: hdfs://prodhadoop/user/ojoqcu/.sparkStaging/application_1481607361601_8315
16/12/22 09:07:27 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 79178 for ojoqcu on ha-hdfs:prodhadoop
16/12/22 09:07:28 INFO metastore: Trying to connect to metastore with URI thrift://l4327pp.sss.com:9083
16/12/22 09:07:29 INFO metastore: Connected to metastore.
16/12/22 09:07:29 INFO YarnSparkHadoopUtil: HBase class not found java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
16/12/22 09:07:29 INFO Client: Source and destination file systems are the same. Not copying hdfs:/lib/spark2_2.0.1.tar.gz
16/12/22 09:07:29 INFO Client: Uploading resource file:/localhome/ojoqcu/code/debug/Rikard/spark-assembly-1.0.jar -> hdfs://prodhadoop/user/ojoqcu/.sparkStaging/application_1481607361601_8315/spark-assembly-1.0.jar
16/12/22 09:07:29 INFO Client: Uploading resource file:/tmp/spark-ff9db580-00db-476e-9086-377c60bc7e2a/__spark_conf__1706674327523194508.zip -> hdfs://prodhadoop/user/ojoqcu/.sparkStaging/application_1481607361601_8315/__spark_conf__.zip
16/12/22 09:07:29 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode
16/12/22 09:07:29 INFO SecurityManager: Changing view acls to: ojoqcu
16/12/22 09:07:29 INFO SecurityManager: Changing modify acls to: ojoqcu
16/12/22 09:07:29 INFO SecurityManager: Changing view acls groups to:
16/12/22 09:07:29 INFO SecurityManager: Changing modify acls groups to:
16/12/22 09:07:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ojoqcu); groups with view permissions: Set(); users  with modify permissions: Set(ojoqcu); groups with modify permissions: Set()
16/12/22 09:07:29 INFO Client: Submitting application application_1481607361601_8315 to ResourceManager
16/12/22 09:07:30 INFO YarnClientImpl: Submitted application application_1481607361601_8315
16/12/22 09:07:31 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:07:31 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: AM container is launched, waiting for AM container to Register with RM
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: dataScientist
         start time: 1482394049862
         final status: UNDEFINED
         tracking URL: http://l4327pp.sss.com:8088/proxy/application_1481607361601_8315/
         user: ojoqcu
16/12/22 09:07:32 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:07:33 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:07:34 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:07:35 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:35 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: 138.106.33.145
         ApplicationMaster RPC port: 0
         queue: dataScientist
         start time: 1482394049862
         final status: UNDEFINED
         tracking URL: http://l4327pp.sss.com:8088/proxy/application_1481607361601_8315/
         user: ojoqcu
16/12/22 09:07:36 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:37 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:38 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:39 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:40 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:41 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:42 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:43 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:44 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:45 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:46 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:47 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:48 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:49 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:50 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:51 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:52 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:53 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:54 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:55 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:56 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:57 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:58 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:07:59 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:00 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:01 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:02 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:03 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:04 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:05 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:06 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:07 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:08 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:09 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:10 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:11 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:12 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:13 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:14 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:15 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:16 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:17 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:18 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:19 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:20 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:21 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:22 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:23 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:24 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:25 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:26 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:27 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:28 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:29 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:30 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:31 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:32 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:33 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:34 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:35 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:36 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:37 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:38 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:39 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:40 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:41 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:42 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:43 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:44 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:45 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:46 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:47 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:48 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:49 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:50 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:51 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:52 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:08:52 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: AM container is launched, waiting for AM container to Register with RM
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: dataScientist
         start time: 1482394049862
         final status: UNDEFINED
         tracking URL: http://l4327pp.sss.com:8088/proxy/application_1481607361601_8315/
         user: ojoqcu
16/12/22 09:08:53 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:08:54 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:08:55 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:08:56 INFO Client: Application report for application_1481607361601_8315 (state: ACCEPTED)
16/12/22 09:08:57 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:57 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: 138.106.33.144
         ApplicationMaster RPC port: 0
         queue: dataScientist
         start time: 1482394049862
         final status: UNDEFINED
         tracking URL: http://l4327pp.sss.com:8088/proxy/application_1481607361601_8315/
         user: ojoqcu
16/12/22 09:08:58 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:08:59 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:00 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:01 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:02 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:03 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:04 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:05 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:06 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:07 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:08 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:09 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:10 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:11 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:12 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:13 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:14 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:15 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:16 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:17 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:18 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:19 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:20 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:21 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:22 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:23 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:24 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:25 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:26 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:27 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:28 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:29 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:30 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:31 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:32 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:33 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:34 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:35 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:36 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:37 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:38 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:39 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:40 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:41 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:42 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:43 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:44 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:45 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:46 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:47 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:48 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:49 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:50 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:51 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:52 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:53 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:54 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:55 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:56 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:57 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:58 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:09:59 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:00 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:01 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:02 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:03 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:04 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:05 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:06 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:07 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:08 INFO Client: Application report for application_1481607361601_8315 (state: RUNNING)
16/12/22 09:10:09 INFO Client: Application report for application_1481607361601_8315 (state: FINISHED)
16/12/22 09:10:09 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 28.0 failed 4 times, most recent failure: Lost task 1.3 in stage 28.0 (TID 59, l4328pp.sss.com): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container marked as failed: container_e63_1481607361601_8315_02_000005 on host: l4328pp.sss.com. Exit status: 143. Diagnostics: Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Killed by external signal

Driver stacktrace:
         ApplicationMaster host: 138.106.33.144
         ApplicationMaster RPC port: 0
         queue: dataScientist
         start time: 1482394049862
         final status: FAILED
         tracking URL: http://l4327pp.sss.com:8088/proxy/application_1481607361601_8315/
         user: ojoqcu
Exception in thread "main" org.apache.spark.SparkException: Application application_1481607361601_8315 finished with failed status
        at org.apache.spark.deploy.yarn.Client.run(Client.scala:1132)
        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1175)
        at org.apache.spark.deploy.yarn.Client.main(Client.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/12/22 09:10:09 INFO ShutdownHookManager: Shutdown hook called
16/12/22 09:10:09 INFO ShutdownHookManager: Deleting directory /tmp/spark-ff9db580-00db-476e-9086-377c60bc7e2a

来自执行作业的其中一个节点的YARN日志：

    2016-12-22 09:10:09,095 WARN  runtime.DefaultLinuxContainerRuntime (DefaultLinuxContainerRuntime.java:launchContainer(107)) - Launch container failed. Exception: 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException: ExitCodeException exitCode=143: 
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:175)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:103)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89)
    at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: ExitCodeException exitCode=143: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:933)
    at org.apache.hadoop.util.Shell.run(Shell.java:844)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1123)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:150)
    ... 9 more

附件是SparkUI的截图。 OutofMemory Heap Space

1 个答案:

答案 0 :(得分：0)

有效的命令：

*最保守，采用最少的群集

/usr/hdp/current/spark2-client/bin/spark-submit --class samples.FuelModel --master yarn --deploy-mode cluster --executor-memory 4g --driver-memory 8g spark-assembly-1.0.jar

所花费的时间太长（40多分钟）是可以接受的，我试着增加号码。核心和执行者，但它没有多大帮助。我认为由于在2个节点上处理小文件（<200MB），开销超过了性能优势。如果有人可以提供任何指示，我会很高兴。

相关问题
最新问题