我从源代码构建了Hadoop 2.7.3,一切都成功了。我正在使用预构建的Spark 2.0二进制文件,支持Hadoop 2.7。当我启动spark-shell时,我收到了这个警告。
16/09/23 14:53:24 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hadoop checknative -a
给了我
16/09/23 14:59:47 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/09/23 14:59:47 WARN zlib.ZlibFactory: Failed to load/initialize native-zlib library
Native library checking:
hadoop: true D:\hadoop-2.7.3\bin\hadoop.dll
zlib: false
snappy: false
lz4: true revision:99
bzip2: false
openssl: false build does not support openssl.
winutils: true D:\hadoop-2.7.3\bin\winutils.exe
16/09/23 14:59:47 INFO util.ExitUtil: Exiting with status 1
我是否必须为所有库获得本机构建?我检查了Hadoop构建指令,但是找不到有关构建其他库的任何信息。
或许我的Spark中有一些错过的配置。但我无法弄清楚是什么。我为Spark设置了这些环境变量:
set HADOOP_HOME=D:/hadoop-2.7.3
set HADOOP_CONF_DIR=%HADOOP_HOME%/etc/hadoop
set SPARK_HOME=D:/spark-2.0.0-bin-hadoop2.7
set HADOOP_COMMON_LIB_NATIVE_DIR=%HADOOP_HOME%/bin
set SPARK_LOCAL_IP=