Question

我的Mac上有自己的开发环境。我在新安装的物理Ubuntu盒子上有一个开发Spark服务器。

在我的代码中，在Mac上运行，我有：

1   SparkConf conf = new SparkConf().setAppName("myapp").setMaster("spark://10.0.100.120:7077");
2   JavaSparkContext javaSparkContext = new JavaSparkContext(conf);
3   javaSparkContext.setLogLevel("WARN");
4   SQLContext sqlContext = new SQLContext(javaSparkContext);
5
6   // Restaurant Data
7   df = sqlContext.read().option("dateFormat", "yyyy-mm-dd").json(source.getLocalStorage());

1）澄清问题：这段代码在我的mac上运行，连接到服务器，但第7行假定文件在我的mac上，而不在服务器上，对吗？

2）在第7行，我得到一个例外：

16-07-10 22:20:04:143 DEBUG  - address: jgp-MacBook-Air.local/10.0.100.100 isLoopbackAddress: false, with host 10.0.100.100 jgp-MacBook-Air.local
16-07-10 22:20:04:240 INFO org.apache.spark.sql.execution.datasources.json.JSONRelation - Listing file:/Users/jgp/Documents/Data/restaurants-data.json on driver
16-07-10 22:20:04:288 DEBUG org.apache.hadoop.util.Shell - Failed to detect a valid hadoop home directory
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
    at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:225)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:250)
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:447)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation.org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd(JSONRelation.scala:98)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4$$anonfun$apply$1.apply(JSONRelation.scala:115)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4$$anonfun$apply$1.apply(JSONRelation.scala:115)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4.apply(JSONRelation.scala:115)
    at org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$4.apply(JSONRelation.scala:109)
    at scala.Option.getOrElse(Option.scala:120)

我是否必须在服务器上安装HADOOP？ - 我想象来自：

java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.

Apache Spark Server安装需要Hadoop吗？没有自动安装？

0 个答案: