这是我的Java代码：
import org.apache.spark.deploy.SparkSubmit;
public class Test{
public static void main(String[] args) {
String[] arg0=new String[]{
"--master","yarn",
"--deploy-mode","client",
"--class","org.apache.spark.examples.SparkPi",
"/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-examples-1.6.2-hadoop2.6.0.jar"
};
SparkSubmit.main(arg0);
}
}
我编译代码如下：
javac -cp ".:/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar" Test.java
然后我像这样设置环境变量：
export HADOOP_CONF_DIR=/opt/yarn_conf
/ opt / yarn_conf从纱线群集上的目录中复制/opt/hadoop-2.6.5/etc/hadoop /
/opt/yarn_conf/yarn-site.xml已使用以下选项配置
<property>
<name>yarn.resourcemanager.address</name>
<value>10.110.16.60:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>10.110.16.60:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>10.110.16.60:8031</value>
</property>
....
最后我运行以下命令
java -cp ".:/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar" Test
然后我得到了这样的信息：
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/02/20 16:22:31 INFO SparkContext: Running Spark version 1.6.2
17/02/20 16:22:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/02/20 16:22:31 INFO SecurityManager: Changing view acls to: root
17/02/20 16:22:31 INFO SecurityManager: Changing modify acls to: root
17/02/20 16:22:31 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
17/02/20 16:22:32 INFO Utils: Successfully started service 'sparkDriver' on port 55141.
17/02/20 16:22:32 INFO Slf4jLogger: Slf4jLogger started
17/02/20 16:22:32 INFO Remoting: Starting remoting
17/02/20 16:22:32 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.110.16.61:37984]
17/02/20 16:22:32 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 37984.
17/02/20 16:22:32 INFO SparkEnv: Registering MapOutputTracker
17/02/20 16:22:32 INFO SparkEnv: Registering BlockManagerMaster
17/02/20 16:22:33 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-109e88d2-ce55-498c-bc0d-78448688fcac
17/02/20 16:22:33 INFO MemoryStore: MemoryStore started with capacity 1088.3 MB
17/02/20 16:22:33 INFO SparkEnv: Registering OutputCommitCoordinator
17/02/20 16:22:33 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/02/20 16:22:33 INFO SparkUI: Started SparkUI at http://10.110.16.61:4040
17/02/20 16:22:33 INFO HttpFileServer: HTTP File server directory is /tmp/spark-2d7ccc96-215f-402a-b425-bf4308bf6a94/httpd-d6478226-76dc-4904-873d-bc9d85e1fe18
17/02/20 16:22:33 INFO HttpServer: Starting HTTP Server
17/02/20 16:22:33 INFO Utils: Successfully started service 'HTTP file server' on port 58022.
17/02/20 16:22:33 INFO SparkContext: Added JAR file:/opt/spark-1.6.2-bin-hadoop2.6/lib/spark-examples-1.6.2-hadoop2.6.0.jar at http://10.110.16.61:58022/jars/spark-examples-1.6.2-hadoop2.6.0.jar with timestamp 1487578953758
17/02/20 16:22:33 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/02/20 16:22:35 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/02/20 16:22:36 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/02/20 16:22:37 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/02/20 16:22:38 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
我该如何更改？认为