Question

我使用的是具有Spark 2.1.0和Scala 2.11.8的Amazon EMR 我想知道当我编译并制作胖Jar时，是否可以使用Scala 2.11.11？

Answer 1

最好能够使用正确的scala版本构建EMR胖罐。这并不难。我想你使用sbt-assembly来制作脂肪罐。在这种情况下，请将此行添加到build.sbt： crossScalaVersions := Seq("2.11.8", "2.11.11")

然后在sbt shell中键入reload以激活此新设置。然后，当你想要在2.11.8上组装一个胖罐时，你可以按照命令键入sbt控制台：

++ 2.11.8
assembly

如果要在本地编译2.11.11或为2.11.11构建胖jar，只需在sbt控制台中切换到该版本即可。例如：

++ 2.11.11
compile

我已经检查过，它对我有用。这是我的sbt控制台的日志：

sbt:TestAssembly> ++ 2.11.8
[info] Setting Scala version to 2.11.8 on 1 projects.
[info] Reapplying settings...
[info] Set current project to TestAssembly (in build file:/ssd2/projects/TestAssembly/)
sbt:TestAssembly> assembly
[info] Including from cache: scala-library-2.11.8.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] Packaging /ssd2/projects/TestAssembly/target/scala-2.11/testStep.jar ...
[info] Done packaging.
[success] Total time: 2 s, completed Nov 29, 2017 12:07:46 PM
sbt:TestAssembly> ++ 2.11.11
[info] Setting Scala version to 2.11.11 on 1 projects.
[info] Reapplying settings...
[info] Set current project to TestAssembly (in build file:/ssd2/projects/TestAssembly/)
sbt:TestAssembly> assembly
[info] Updating {file:/ssd2/projects/TestAssembly/}testassembly...
[info] Done updating.
[info] Compiling 2 Scala sources to /ssd2/projects/TestAssembly/target/scala-2.11/classes ...
[info] Non-compiled module 'compiler-bridge_2.11' for Scala 2.11.11. Compiling...
[info]   Compilation completed in 13.098s.
[info] Done compiling.
[info] Including: scala-library-2.11.11.jar
[info] Checking every *.class/*.jar file's SHA-1.
[info] Merging files...
[warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
[warn] Strategy 'discard' was applied to a file
[info] SHA-1: b0a0cc7ec39ca359187a0b24f303c376b1f2d71f
[info] Packaging /ssd2/projects/TestAssembly/target/scala-2.11/testStep.jar ...
[info] Done packaging.
[success] Total time: 17 s, completed Nov 29, 2017 12:10:37 PM
sbt:TestAssembly>

正如您所看到的那样，每种情况下都包含适当的scala库jar。

当我制作胖Jar时，可以在EMR中使用不同的Scala版本吗？

1 个答案: