将tensorflow导入Scala

时间:2018-02-15 17:01:09

标签: scala tensorflow

我在我的OSX上安装了Tensorflow for Scala,虽然一切看起来都很好,但在尝试运行这样的简单示例时,我有一个NoClassDefFoundError -

   import org.platanios.tensorflow.api.Tensor
   val tensor = Tensor( 1.2, 4.5)

给出:

java.lang.NoClassDefFoundError: Could not initialize class org.platanios.tensorflow.api.package$
    at #worksheet#.tensor$lzycompute(testone.sc:3)
    at #worksheet#.tensor(testone.sc:3)
    at #worksheet#.get$$instance$$tensor(testone.sc:3)
    at A$A16$.main(testone.sc:17)
    at A$A16.main(testone.sc)
    at #worksheet#.#worksheet#(testone.sc)

Jupyter Notebook和IntelliJ工作表上的类似错误。我的build.sbt:

scalaVersion := "2.12.4"

resolvers += Resolver.sonatypeRepo("snapshots")
libraryDependencies += "org.platanios" %% "tensorflow" % "0.1.2-SNAPSHOT"

libraryDependencies += "org.platanios" %% "tensorflow" % "0.1.2-SNAPSHOT" classifier "darwin-cpu-x86_64"

1 个答案:

答案 0 :(得分:1)

问题可能是由于scala_tensorflow jar包含的libtensorflow_jni.so缺少依赖库

找到丢失的库:

sbt console

然后从scala shell导入tensorflow api:

scala> import org.platanios.tensorflow.api._
scala> val tensor = Tensor.zeros(INT32, Shape(2, 5))
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/angelo/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/angelo/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No appenders could be found for logger (TensorFlow Native).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.lang.UnsatisfiedLinkError: /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_jni.so: libcusolver.so.9.0: cannot open shared object file: No such file or directory
  at java.lang.ClassLoader$NativeLibrary.load(Native Method)
  at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
  at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
  at java.lang.Runtime.load0(Runtime.java:809)
  at java.lang.System.load(System.java:1086)
  at org.platanios.tensorflow.jni.TensorFlow$$anonfun$load$3.apply(TensorFlow.scala:95)
  at org.platanios.tensorflow.jni.TensorFlow$$anonfun$load$3.apply(TensorFlow.scala:93)
  at scala.Option.foreach(Option.scala:257)
  at org.platanios.tensorflow.jni.TensorFlow$.load(TensorFlow.scala:93)
  at org.platanios.tensorflow.jni.TensorFlow$.<init>(TensorFlow.scala:155)
  at org.platanios.tensorflow.jni.TensorFlow$.<clinit>(TensorFlow.scala)
  at org.platanios.tensorflow.jni.Tensor$.<init>(Tensor.scala:24)
  at org.platanios.tensorflow.jni.Tensor$.<clinit>(Tensor.scala)
  at org.platanios.tensorflow.api.tensors.Context$.apply(Context.scala:50)
  at org.platanios.tensorflow.api.package$.<init>(package.scala:89)
  at org.platanios.tensorflow.api.package$.<clinit>(package.scala)
  ... 40 elided

你可以在另一个终端上使用ldd检查所有缺少的库(linux中的ldd或macosx上的otool -L):

ldd /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_jni.so
/tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_jni.so: /usr/lib/libcublas.so.9.0: version `libcublas.so.9.0' not found (required by /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow.so)
/tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_jni.so: /usr/lib/libcublas.so.9.0: version `libcublas.so.9.0' not found (required by /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_framework.so)
        linux-vdso.so.1 =>  (0x00007ffc00d85000)
        libdlfaker.so => /usr/lib/x86_64-linux-gnu/libdlfaker.so (0x00007f0cd05d8000)
        librrfaker.so => /usr/lib/x86_64-linux-gnu/librrfaker.so (0x00007f0cd033d000)
        libtensorflow.so => /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow.so (0x00007f0cc8023000)
        libtensorflow_framework.so => /tmp/tensorflow_scala_native_libraries3327494822622243889/libtensorflow_framework.so (0x00007f0cc714b000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0cc6dc9000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0cc6bb3000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0cc67e9000)
        libGL.so.1 => /usr/lib/nvidia-390/libGL.so.1 (0x00007f0cc64ac000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0cc62a8000)
        libturbojpeg.so.0 => /usr/lib/x86_64-linux-gnu/libturbojpeg.so.0 (0x00007f0cc6047000)
        libXv.so.1 => /usr/lib/x86_64-linux-gnu/libXv.so.1 (0x00007f0cc5e42000)
        libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f0cc5b08000)
        libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f0cc58f6000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0cc56d9000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0cc53d0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f0cd0a66000)
        libcublas.so.9.0 => /usr/lib/libcublas.so.9.0 (0x00007f0cc1caf000)
        libcusolver.so.9.0 => not found
        libcudart.so.9.0 => not found
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f0cc1a8d000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0cc1885000)
        libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x00007f0cc0ce5000)
        libcudnn.so.7 => /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudnn.so.7 (0x00007f0cafd54000)
        libcufft.so.9.0 => not found
        libcurand.so.9.0 => not found
        libcudart.so.9.0 => not found
        libnvidia-tls.so.390.30 => /usr/lib/nvidia-390/tls/libnvidia-tls.so.390.30 (0x00007f0cafb50000)
        libnvidia-glcore.so.390.30 => /usr/lib/nvidia-390/libnvidia-glcore.so.390.30 (0x00007f0cadd50000)
        libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f0cadb2e000)
        libnvidia-fatbinaryloader.so.390.30 => /usr/lib/nvidia-390/libnvidia-fatbinaryloader.so.390.30 (0x00007f0cad8e2000)
        libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f0cad6de000)
        libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f0cad4d8000)

在我的计算机上,由于我安装了cuda-9.1,因此rutime链接过程无法解析libcusolver.so.9.0

要使其正常工作,我必须compile tensorflow如下:

git clone https://github.com/tensorflow/tensorflow.git 
cd tensorflow

./configure

bazel build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --config=opt //tensorflow:libtensorflow.so

将库复制到LD_LIBRARY_PATH中的路径:

sudo cp bazel-bin/tensorflow/libtensorflow.so /usr/local/lib
sudo cp bazel-bin/tensorflow/libtensorflow_framework.so /usr/local/lib

最后我能够编译tensorflow_scala项目:

sbt compile

现在从tensorflow_scala项目我可以运行sbt console并且它可以运行:

scala> import org.platanios.tensorflow.api._
import org.platanios.tensorflow.api._

scala> val tensor = Tensor.zeros(INT32, Shape(2, 5))
2018-02-16 17:08:26.184 [run-main-0] INFO  TensorFlow Native - Extracting the 'tensorflow_jni' native library to /tmp/tensorflow_scala_native_libraries8283851378265055495/libtensorflow_jni.so.
2018-02-16 17:08:26.188 [run-main-0] INFO  TensorFlow Native - Copied 645872 bytes to /tmp/tensorflow_scala_native_libraries8283851378265055495/libtensorflow_jni.so.
2018-02-16 17:08:26.254 [run-main-0] INFO  TensorFlow Native - Extracting the 'tensorflow_ops' native library to /tmp/tensorflow_scala_native_libraries8283851378265055495/libtensorflow_ops.so.
2018-02-16 17:08:26.254 [run-main-0] INFO  TensorFlow Native - Copied 78232 bytes to /tmp/tensorflow_scala_native_libraries8283851378265055495/libtensorflow_ops.so.
2018-02-16 17:08:26.449239: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-02-16 17:08:26.449483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1331] Found device 0 with properties: 
name: GeForce GTX 1050 major: 6 minor: 1 memoryClockRate(GHz): 1.493
pciBusID: 0000:01:00.0
totalMemory: 3,95GiB freeMemory: 1,53GiB
2018-02-16 17:08:26.449498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1410] Adding visible gpu devices: 0
2018-02-16 17:08:26.670568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-02-16 17:08:26.670601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0 
2018-02-16 17:08:26.670610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N 
2018-02-16 17:08:26.670698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1021] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1287 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-02-16 17:08:26.698355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1410] Adding visible gpu devices: 0
2018-02-16 17:08:26.698399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-02-16 17:08:26.698408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0 
2018-02-16 17:08:26.698414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N 
2018-02-16 17:08:26.698518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1021] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 224 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050, pci bus id: 0000:01:00.0, compute capability: 6.1)
tensor: org.platanios.tensorflow.api.tensors.Tensor = INT32[2, 5]

使用本地构建的tensorflow_scala:

sbt

sbt:TensorFlow for Scala> + publishLocal

将jar放入〜/ .ivy2 / local / org.platanios;然后你可以将jar添加到你的sbt项目中,例如:

libraryDependencies ++= { Seq("org.platanios" %% "tensorflow" % "0.1.2-SNAPSHOT") }