Question

我们正在尝试从EMR写入DSE图（cassandra），并不断得到这些错误。我的JAR是带有byos依赖项的阴影罐子。任何帮助将不胜感激。

java.lang.UnsatisfiedLinkError: org.apache.cassandra.utils.NativeLibraryLinux.getpid()J
    at org.apache.cassandra.utils.NativeLibraryLinux.getpid(Native Method)
    at org.apache.cassandra.utils.NativeLibraryLinux.callGetpid(NativeLibraryLinux.java:124)
    at org.apache.cassandra.utils.NativeLibrary.getProcessID(NativeLibrary.java:429)
    at org.apache.cassandra.utils.UUIDGen.hash(UUIDGen.java:386)
    at org.apache.cassandra.utils.UUIDGen.makeNode(UUIDGen.java:367)
    at org.apache.cassandra.utils.UUIDGen.makeClockSeqAndNode(UUIDGen.java:300)
    at org.apache.cassandra.utils.UUIDGen.<clinit>(UUIDGen.java:41)
    at com.datastax.bdp.graph.spark.sql.vertex.SimpleVertexIdAssigner$.simpleEdgeId(SimpleVertexIdAssigner.scala:19)
    at com.datastax.bdp.graph.spark.graphframe.DseGraphFrame$$anonfun$3.apply(DseGraphFrame.scala:417)
    at com.datastax.bdp.graph.spark.graphframe.DseGraphFrame$$anonfun$3.apply(DseGraphFrame.scala:416)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
    at org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:131)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:220)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:298)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

19/04/26 12:55:49 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 18, ip-10-69-16-79.vpc.internal, executor 1): java.lang.NoClassDefFoundError: Could not initialize class org.apache.cassandra.utils.UUIDGen
    at com.datastax.bdp.graph.spark.sql.vertex.SimpleVertexIdAssigner$.simpleEdgeId(SimpleVertexIdAssigner.scala:19)
    at com.datastax.bdp.graph.spark.graphframe.DseGraphFrame$$anonfun$3.apply(DseGraphFrame.scala:417)
    at com.datastax.bdp.graph.spark.graphframe.DseGraphFrame$$anonfun$3.apply(DseGraphFrame.scala:416)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)

Answer 1

通常，当使用noexec属性安装临时目录时会发生此类错误，该属性阻止加载Java驱动程序使用的本机库。通常的解决方法是将Java指向带有-Djava.io.tmpdir=...标志的临时文件的另一个位置-不应使用noexec标志挂载此位置。

P.S。不幸的是，我对EMR并不了解

Answer 2

原来是JNA问题。添加了JNA依赖项作为阴影jar的一部分，并且可以正常工作。

<dependency>
        <groupId>net.java.dev.jna</groupId>
        <artifactId>jna</artifactId>
        <version>4.2.2</version>
</dependency>

从EMR写入DSE图

2 个答案: