hdisght上的Spark作业服务器安装

时间：2017-04-01 21:06:05

标签： azure apache-spark hdinsight

我创建了一个azure hdinsight spark集群，我正在尝试在headnode上安装Spark Job Server。我已经进入了headnode，这是我正在遵循的步骤

其HDI 3.5与Spark 1.6.3

  1. 安装SBT：

    2. echo“deb https://dl.bintray.com/sbt/debian /”| sudo tee -a /etc/apt/sources.list.d/sbt.list

    sudo apt-key adv --keyserver hkp：//keyserver.ubuntu.com：80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823

    sudo apt-get update

    sudo apt-get install sbt

    1. git clone https://github.com/spark-jobserver/spark-jobserver.git

      2. 主人是火花1.6.3，所以我不换树枝。

      1. 进入spark-observer目录

      2. sbt assembly 此时我遇到了这样的错误

        3. * 

        [warn] Merging 'reference.conf' with strategy 'concat'
[error] 1 error was encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/home/kmk/.ivy2/cache/io.netty/netty-all/jars/netty-all-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.0.37.Final.jar:META-INF/io.netty.versions.properties
/home/kmk/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.0.37.Final.jar:META-INF/io.netty.versions.properties
    at sbtassembly.Assembly$.applyStrategies(Assembly.scala:140)
    at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
    at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
    at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
    at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
    at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
    at sbtassembly.Assembly$.apply(Assembly.scala:83)
    at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:240)
    at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:237)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
java.lang.RuntimeException: Building python API received non-zero exit code 1
    at scala.sys.package$.error(package.scala:27)
    at PythonTasks$.buildPythonTask(PythonTasks.scala:25)
    at $b9e935155022d705b7b0$$anonfun$jobServerPythonSettings$5.apply(build.sbt:110)
    at $b9e935155022d705b7b0$$anonfun$jobServerPythonSettings$5.apply(build.sbt:110)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:237)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[error] (root/*:assembly) deduplicate: different file contents found in the following:
[error] /home/kmk/.ivy2/cache/io.netty/netty-all/jars/netty-all-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-handler/jars/netty-handler-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-buffer/jars/netty-buffer-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-common/jars/netty-common-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-transport/jars/netty-transport-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] /home/kmk/.ivy2/cache/io.netty/netty-codec/jars/netty-codec-4.0.37.Final.jar:META-INF/io.netty.versions.properties
[error] (job-server-python/*:buildPython) Building python API received non-zero exit code 1
[error] Total time: 151 s, completed Apr 1, 2017 3:41:54 PM

        *

        我在这里做错了什么？

1 个答案:

答案 0 :(得分：0)

根据您的错误信息，根据我的经验，我认为问题是由这些罐子的不同版本的冲突造成的，这些罐子由相同的包名称命名。所以通常的方法是尝试删除这些重复的jar。或者您可以参考sbt的官方文档Excluding JARs and files来通过合并策略来解决依赖关系，有一个SO帖子Resolving Dependencies in creating JAR through SBT assembly可以帮助您作为参考。

同时，根据spark-job-server进行sbt job-server/assembly部署，{{1}}操作似乎没有必要，这取决于您当前的环境。

希望它有所帮助。

相关问题
最新问题