Question

当我在Hadoop上运行Apache Kylin时，遇到以下与Hadoop MapReduce相关的错误：

2019-03-20 08:06:00,193 ERROR [main] org.apache.kylin.engine.mr.KylinMapper: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z     
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)     
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)     
at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:136)     
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)     
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:168)     
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1304)     
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1192)     
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1552)     
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:289)     
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:542)     
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64) 
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)     
at org.apache.hadoop.mapreduce.lib.output.LazyOutputFormat$LazyRecordWriter.write(LazyOutputFormat.java:113)     
at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:468)

我想原因是Hadoop无法找到libsnappy.so *本机库。我已在线搜索解决方案。在此link之后，我已经在相应的xml文件中添加了以下属性并重新启动了服务：

# For HDFS core-site.xml
<property>
  <name>io.compression.codecs</name>
  <value>
    org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec
  </value>
</property>
# For MapReduce2 mapred-site.xml
<property>
  <name>mapreduce.admin.user.env</name>
  <value>LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native</value>
</propert
<property> 
  <name>mapreduce.map.output.compress</name>  
  <value>true</value>
</property>
<property>
  <name>mapred.map.output.compress.codec</name>  
  <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
  <name>mapreduce.output.fileoutputformat.compress</name>
  <value>true</value> 
</property> 
<property> 
  <name>mapreduce.output.fileoutputformat.compress.codec</name>
  <value>org.apache.hadoop.io.compress.SnappyCodec</value> 
</property>

但是，它没有用。因此，我深入研究了YARN的日志。我发现在launch_container.sh部分中，它具有以下命令：

export PWD="/hadoop/yarn/local/usercache/root/appcache/application_1553049994285_0013/container_e04_1553049994285_0013_01_000005"
# ...omit other commands
export LD_LIBRARY_PATH="$PWD"

我认为此命令是错误的，因为libsnappy.so *的真实路径不是$ PWD。

LD_LIBRARY_PATH=/usr/hdp/${hdp.version}/hadoop/lib/native

正如您所看到的，我已经将LD_LIBRARY_PATH设置为指向mapred-site.xml文件中的真实路径。为什么yarn仍然使用$ PWD？

此外，我添加了一条日志消息，如下所示。这样可以确保确实错误地设置了env LD_LIBRARY_PATH。那我该如何解决这个问题呢？

2019-03-20 08:06:00,044 INFO [main] org.apache.kylin.engine.mr.KylinMapper: linyanwen[from map]: /hadoop/yarn/local/usercache/root/appcache/application_1553049994285_0039/container_e04_1553049994285_0039_01_000005

谢谢！

Hadoop压缩错误：java.lang.UnsatisfiedLinkError：org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy（）Z

0 个答案: