Question

我在Ubuntu 16.04上运行Spark 2.1.0，Hive 2.1.1和Hadoop 2.7.3。

我从github下载Spark项目并构建＆＃34;没有hadoop＆＃34;版本：

./ dev / make-distribution.sh --name＆＃34; hadoop2-without-hive＆＃34; --tgz ＆＃34; -Pyarn，Hadoop的提供，Hadoop的2.7，镶木地板，提供＆＃34;

当我运行./sbin/start-master.sh时，我得到以下异常：

 Spark Command: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp /home/server/spark/conf/:/home/server/spark/jars/*:/home/server/hadoop/etc/hadoop/:/home/server/hadoop/share/hadoop/common/lib/:/home/server/hadoop/share/hadoop/common/:/home/server/hadoop/share/hadoop/mapreduce/:/home/server/hadoop/share/hadoop/mapreduce/lib/:/home/server/hadoop/share/hadoop/yarn/:/home/server/hadoop/share/hadoop/yarn/lib/ -Xmx1g org.apache.spark.deploy.master.Master --host ThinkPad-W550s-Lab --port 7077 --webui-port 8080
 ========================================
 Error: A JNI error has occurred, please check your installation and try again
 Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
     at java.lang.Class.getDeclaredMethods0(Native Method)
     at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
     at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
     at java.lang.Class.getMethod0(Class.java:3018)
     at java.lang.Class.getMethod(Class.java:1784)
     at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
     at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
 Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
     ... 7 more

我根据帖子Where are hadoop jar files in hadoop 2?

编辑SPARK_DIST_CLASSPATH

export SPARK_DIST_CLASSPATH=~/hadoop/share/hadoop/common/lib:~/hadoop/share/hadoop/common:~/hadoop/share/hadoop/mapreduce:~/hadoop/share/hadoop/mapreduce/lib:~/hadoop/share/hadoop/yarn:~/hadoop/share/hadoop/yarn/lib

但我仍然遇到同样的错误。我可以看到slf4j jar文件在~/hadoop/share/hadoop/common/lib下。

我该如何解决此错误？

谢谢！

Answer 1

“Hadoop free”构建需要修改SPARK_DIST_CLASSPATH以包含Hadoop的包jar。最方便的地方是在conf / spark-env.sh中添加一个条目：
export SPARK_DIST_CLASSPATH = $（/ path / to / hadoop / bin / hadoop classpath）
检查此https://spark.apache.org/docs/latest/hadoop-provided.html

没有Hadoop的Spark：无法启动

1 个答案: