当我运行java程序在yarn上提交spark-app时,我得到“Retrying connect to server:0.0.0.0/0.0.0.0:8032”info

时间:2017-02-20 08:45:49

标签: java hadoop apache-spark yarn resourcemanager

这是我的Java代码:

import org.apache.spark.deploy.SparkSubmit;

public class Test{
    public static void main(String[] args) {  
        String[] arg0=new String[]{  
            "--master","yarn",
            "--deploy-mode","client",
            "--class","org.apache.spark.examples.SparkPi",  
            "/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-examples-1.6.2-hadoop2.6.0.jar"
        };  
        SparkSubmit.main(arg0);  
    }
}

我编译代码如下:

javac -cp ".:/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar" Test.java

然后我像这样设置环境变量:

export HADOOP_CONF_DIR=/opt/yarn_conf

/ opt / yarn_conf从纱线群集上的目录中复制/opt/hadoop-2.6.5/etc/hadoop /

/opt/yarn_conf/yarn-site.xml已使用以下选项配置

<property>
  <name>yarn.resourcemanager.address</name>
  <value>10.110.16.60:8032</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address</name>
  <value>10.110.16.60:8030</value>
</property>
<property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>10.110.16.60:8031</value>
</property>
....

最后我运行以下命令

java -cp ".:/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar" Test

然后我得到了这样的信息:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/02/20 16:22:31 INFO SparkContext: Running Spark version 1.6.2
17/02/20 16:22:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/02/20 16:22:31 INFO SecurityManager: Changing view acls to: root
17/02/20 16:22:31 INFO SecurityManager: Changing modify acls to: root
17/02/20 16:22:31 INFO SecurityManager: SecurityManager: authentication    disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
17/02/20 16:22:32 INFO Utils: Successfully started service 'sparkDriver' on port 55141.
17/02/20 16:22:32 INFO Slf4jLogger: Slf4jLogger started
17/02/20 16:22:32 INFO Remoting: Starting remoting
17/02/20 16:22:32 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.110.16.61:37984]
17/02/20 16:22:32 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 37984.
17/02/20 16:22:32 INFO SparkEnv: Registering MapOutputTracker
17/02/20 16:22:32 INFO SparkEnv: Registering BlockManagerMaster
17/02/20 16:22:33 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-109e88d2-ce55-498c-bc0d-78448688fcac
17/02/20 16:22:33 INFO MemoryStore: MemoryStore started with capacity 1088.3 MB
17/02/20 16:22:33 INFO SparkEnv: Registering OutputCommitCoordinator
17/02/20 16:22:33 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/02/20 16:22:33 INFO SparkUI: Started SparkUI at http://10.110.16.61:4040
17/02/20 16:22:33 INFO HttpFileServer: HTTP File server directory is /tmp/spark-2d7ccc96-215f-402a-b425-bf4308bf6a94/httpd-d6478226-76dc-4904-873d-bc9d85e1fe18
17/02/20 16:22:33 INFO HttpServer: Starting HTTP Server
17/02/20 16:22:33 INFO Utils: Successfully started service 'HTTP file server' on port 58022.
17/02/20 16:22:33 INFO SparkContext: Added JAR file:/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-examples-1.6.2-hadoop2.6.0.jar at http://10.110.16.61:58022/jars/spark-examples-1.6.2-hadoop2.6.0.jar with timestamp 1487578953758
17/02/20 16:22:33 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/02/20 16:22:35 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/02/20 16:22:36 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/02/20 16:22:37 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/02/20 16:22:38 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

我该如何更改?认为

0 个答案:

没有答案