Question

我在一个主节点和一个核心节点上使用Spark on EMR，我正面临着以下问题：

java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.

我将java.library.path设置为 / usr / lib / hadoop / lib / native /：/ usr / lib / hadoop-lzo / lib / native /

checknative在主节点和核心节点上给出了这个：

16/04/28 15:03:27 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/04/28 15:03:27 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /usr/lib/hadoop/lib/native/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so

EMR VERSION：emr-4.6.0

Hadoop发布：亚马逊2.7.2

应用程序：Spark 1.6.1

Answer 1

我终于找到了解决方案，我设置了LD_LIBRARY_PATH，它现在正常工作!!

export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native/

我在所有节点上执行了此操作。我知道它会帮助某人:)！

EMR上的Snappy不可用错误

1 个答案: