Question

我尝试使用yarn运行OLAP和批量加载。版本为janusgrapg-0.3.2，spark-2.2.1

为了在janusgraph中启动spark-yarn，我将jar的jar复制到$ JANUSGRAPH_HOME / lib中。

hadoop-graphson.properties：

gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphReader=org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONInputFormat
gremlin.hadoop.graphWriter=org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONOutputFormat
gremlin.hadoop.inputLocation=data/tinkerpop-modern.json
gremlin.hadoop.outputLocation=output
gremlin.hadoop.jarsInDistributedCache=true
gremlin.vertexProgram=org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram
giraph.minWorkers=2
giraph.maxWorkers=2
spark.master=yarn
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.kryo.registrator=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoRegistrator

gremlin控制台中的命令：

graph = GraphFactory.open('/Users/lwh/dev/janusgraph-0.3.2-SNAPSHOT-hadoop2/conf/hadoop-graph/hadoop-graphson.properties')
blvp = BulkLoaderVertexProgram.build().bulkLoader(OneTimeBulkLoader).writeGraph('/Users/lwh/dev/janusgraph-0.3.2-SNAPSHOT-hadoop2/conf/janusgraph-cql-es.properties').create(graph);graph.compute(SparkGraphComputer).program(blvp).submit().get()

纱线结果：

Application application_1547728549460_0008 failed 2 times due to AM Container for appattempt_1547728549460_0008_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://lwhdeMacBook-Pro.local:8088/cluster/app/application_1547728549460_0008Then, click on links to logs of each attempt.
Diagnostics: Resource file:/Users/lwh/.sparkStaging/application_1547728549460_0008/__spark_libs__7746382379788050915.zip changed on src filesystem (expected 1547732288000, was 1547732288958
java.io.IOException: Resource file:/Users/lwh/.sparkStaging/application_1547728549460_0008/__spark_libs__7746382379788050915.zip changed on src filesystem (expected 1547732288000, was 1547732288958
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:255)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Failing this attempt. Failing the application.

Answer 1

可能是由于版本匹配引起的，例如文档 here ，我将spark 2.0.2和janusgraph 0.3.2一起部署，当我更改为1.3.X的spark版本时，它总是抛出异常。，效果很好！

如何在janusgraph中使用yarn来运行OLAP

1 个答案: