Spark - WARN LoadSnappy:未加载Snappy本机库

时间:2015-04-26 21:32:38

标签: pyspark snappy

尝试从Spark Summit 2014开始练习。在终端中运行命令时,我一直收到以下信息:

Spark assembly has been built with Hive, including Datanucleus jars on classpath
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; 
support was removed in 8.0
15/04/26 14:21:05 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
15/04/26 14:21:05 WARN LoadSnappy: Snappy native library not loaded

我在网上发现,MAC上会出现第一个NativeCodeLoader警告。那第二个怎么样?我怎样才能加载Snappy?

谢谢!

编辑:按原样,代码运行,生成上面的输出,没有别的。

2 个答案:

答案 0 :(得分:0)

如果您运行的是简单的Map Reduce程序,可以通过安装snappy库并将其放在类路径中来避免该警告。你可以从这里下载jar https://www.versioneye.com/java/org.xerial.snappy:snappy-java/1.1.1.2

如果您正在使用Maven项目,则必须在pom.xml中添加此依赖项

<dependency>
  <groupId>org.xerial.snappy</groupId>
  <artifactId>snappy-java</artifactId>
  <version>1.1.1.7</version>
  <type>jar</type>
  <scope>compile</scope>
</dependency>

希望这对你有用。

答案 1 :(得分:-1)

在OSX环境中预计会出现这些警告。确认您已正确遵循教程中的步骤。要查看任何输出,您需要更新原始MovieLensALS.scala文件。

在&#34; //代码下面添加以下代码&#34;评价。

val numRatings = ratings.count
val numUsers = ratings.map(_._2.user).distinct.count
val numMovies = ratings.map(_._2.product).distinct.count

println("Got " + numRatings + " ratings from "
  + numUsers + " users on " + numMovies + " movies.")