如果提交了如下所示的内置示例,则我在独立集群中设置了火花。
./bin/spark-submit \
--class JavaWordCount \
--master spark://192.168.77.130:7077 \
/root/spark/examples/jars/spark-examples_2.11-2.4.1.jar \
/tmp/abc.txt
然后,工作节点中的日志如下所示。
19/04/21 09:07:54 INFO DriverRunner: Launch Command: "/root/jdk1.8.0_191/bin/java" "-cp" "/root/spark/conf/:/root/spark/jars/*" "-Xmx1024M" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://192.168.77.130:7077" "-Dspark.app.name=JavaWordCount" "-Dspark.jars=file:/root/spark/examples/jars/spark-examples_2.11-2.4.1.jar" "-Dspark.driver.supervise=true" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.77.131:34646" "/root/spark/work/driver-20190421090754-0000/spark-examples_2.11-2.4.1.jar" "JavaWordCount" "/tmp/abc.txt"
19/04/21 09:07:57 INFO DriverRunner: Command exited with status 1, re-launching after 1 s.
19/04/21 09:07:58 INFO DriverRunner: Launch Command: "/root/jdk1.8.0_191/bin/java" "-cp" "/root/spark/conf/:/root/spark/jars/*" "-Xmx1024M" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://192.168.77.130:7077" "-Dspark.app.name=JavaWordCount" "-Dspark.jars=file:/root/spark/examples/jars/spark-examples_2.11-2.4.1.jar" "-Dspark.driver.supervise=true" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.77.131:34646" "/root/spark/work/driver-20190421090754-0000/spark-examples_2.11-2.4.1.jar" "JavaWordCount" "/tmp/abc.txt"
19/04/21 09:08:00 INFO DriverRunner: Command exited with status 1, re-launching after 2 s.
19/04/21 09:08:02 INFO DriverRunner: Launch Command: "/root/jdk1.8.0_191/bin/java" "-cp" "/root/spark/conf/:/root/spark/jars/*" "-Xmx1024M" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://192.168.77.130:7077" "-Dspark.app.name=JavaWordCount" "-Dspark.jars=file:/root/spark/examples/jars/spark-examples_2.11-2.4.1.jar" "-Dspark.driver.supervise=true" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.77.131:34646" "/root/spark/work/driver-20190421090754-0000/spark-examples_2.11-2.4.1.jar" "JavaWordCount" "/tmp/abc.txt"
19/04/21 09:08:05 INFO DriverRunner: Command exited with status 1, re-launching after 4 s.
19/04/21 09:08:09 INFO DriverRunner: Launch Command: "/root/jdk1.8.0_191/bin/java" "-cp" "/root/spark/conf/:/root/spark/jars/*" "-Xmx1024M" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://192.168.77.130:7077" "-Dspark.app.name=JavaWordCount" "-Dspark.jars=file:/root/spark/examples/jars/spark-examples_2.11-2.4.1.jar" "-Dspark.driver.supervise=true" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.77.131:34646" "/root/spark/work/driver-20190421090754-0000/spark-examples_2.11-2.4.1.jar" "JavaWordCount" "/tmp/abc.txt"
19/04/21 09:08:11 INFO DriverRunner: Command exited with status 1, re-launching after 8 s.
19/04/21 09:08:19 INFO DriverRunner: Launch Command: "/root/jdk1.8.0_191/bin/java" "-cp" "/root/spark/conf/:/root/spark/jars/*" "-Xmx1024M" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://192.168.77.130:7077" "-Dspark.app.name=JavaWordCount" "-Dspark.jars=file:/root/spark/examples/jars/spark-examples_2.11-2.4.1.jar" "-Dspark.driver.supervise=true" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.77.131:34646" "/root/spark/work/driver-20190421090754-0000/spark-examples_2.11-2.4.1.jar" "JavaWordCount" "/tmp/abc.txt"
19/04/21 09:08:22 INFO DriverRunner: Command exited with status 1, re-launching after 16 s.
19/04/21 09:08:38 INFO DriverRunner: Launch Command: "/root/jdk1.8.0_191/bin/java" "-cp" "/root/spark/conf/:/root/spark/jars/*" "-Xmx1024M" "-Dspark.submit.deployMode=cluster" "-Dspark.master=spark://192.168.77.130:7077" "-Dspark.app.name=JavaWordCount" "-Dspark.jars=file:/root/spark/examples/jars/spark-examples_2.11-2.4.1.jar" "-Dspark.driver.supervise=true" "-Dspark.rpc.askTimeout=10s" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.77.131:34646" "/root/spark/work/driver-20190421090754-0000/spark-examples_2.11-2.4.1.jar" "JavaWordCount" "/tmp/abc.txt"
19/04/21 09:08:41 INFO DriverRunner: Command exited with status 1, re-launching after 32 s.