将DataFrame写入HDFS，拒绝连接

Question

我正在尝试从精通Apache Spark 2.x书中运行示例。

scala> val df = sc.parallelize(Array(1,2,3)).toDF
df: org.apache.spark.sql.DataFrame = [value: int]

我是Spark世界的新手，但我想应该将数据帧保存到HDFS

scala> df.write.json("hdfs://localhost:9000/tmp/account.json")
java.net.ConnectException: Call From miki/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

我与dfsadmin检查了

 hadoop dfsadmin -safemode enter
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

safemode: FileSystem file:/// is not an HDFS file system

jps输出

miki@miki:~$ jps
13798 Jps
10906 SparkSubmit

该如何解决？

Answer 1

根据jps输出，您没有运行读写HDFS（namenode，datanode，resourcemanager）所需的必要Hadoop守护程序。确保在计算机上运行start-yarn和start-dfs，以使HDFS正常运行。