Question

我有一个非常简单的Scala程序，它是获取从数据库中提取的字符串的哈希值。我必须使用的散列函数是仅由C ++实现的Google生产的cityhash64withseed。所以我的解决方案是构建一个C ++ libCityhashNative.so，由我的Scala spark程序通过JNI调用。该.so将返回我传入的字符串的哈希值。

我将libCityhashNative.so打包在Runnable jar tdw_spark-1.0-all.jar中。
从jar中提取它并将其写入服务器的临时路径。
通过System.load(tmp.getAbsolutePath())加载。

我的代码是这样的：

    package com.tencent.omg.tasks.jni
    class  CityhashNative {
    // --- Native methods
    @native def intMethod(n: Int): Int
    @native def getCityHash64(strToHash: String, length: Int, ddwSeed: Int): Int
    }

    object CityhashNative {
        val path="/video/libCityhashNative.so"
        val prefix = "libCityhashNative"
        val suffix = ".so"

        //create tmp file on server with file prefix and suffix
        val tmp = File.createTempFile(prefix, suffix)

        var buff = new Array[Byte](1024)
        val len : Int = 0
        val out = new FileOutputStream(tmp)
        //load libCityhashNative.so from Jar to inputStream
        val in = getClass.getResourceAsStream(path)
        val reader = new BufferedInputStream(in)

        //write libCityhashNative.so to server location tmp via outputStream
        while(reader.read(buff)>0){
          out.write(buff)
          buff = new Array[Byte](1024)
        }

        System.load(tmp.getAbsolutePath())
        val chn = new CityhashNative
        val hashValue = chn.getCityHash64("XXXX", 9, 5)
}

连接的东西是当我运行罐子时

spark-submit --master local  --class com.tencent.omg.tasks.jni.CityhashNative tdw_spark-1.0-all.jar

我将得到以下错误：

Exception in thread "main" java.lang.UnsatisfiedLinkError: com.tencent.omg.tasks.jni.CityhashNative.getCityHash64(Ljava/lang/String;II)I
    at com.tencent.omg.tasks.jni.CityhashNative.getCityHash64(Native Method)
    at com.tencent.omg.tasks.jni.CityhashNative$.main(CityhashNative.scala:86)
    at com.tencent.omg.tasks.jni.CityhashNative.main(CityhashNative.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

但是如果我是

scala -cp . CityhashNative

有效！

所以我有点困惑在生成jar的地方做错了。还是原因是.so无法成功加载？

任何建议将不胜感激。

PS：我确信libCityhashNative.so已成功从jar中提取并写入服务器临时路径。

执行Scala可运行jar时出现java.lang.UnsatisfiedLinkError

0 个答案: