Question

我正在使用Spark 2.2（使用Scala 2.11.8构建）将我的数据编入索引到ElasticSearch 5.4.2。

ElasticSearch：

我的项目使用这个pom.xml：

  <dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch-hadoop</artifactId>
        <version>5.4.2</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-spark-20_2.11</artifactId>
        <version>5.4.2</version>
    </dependency>

当我开始工作时，我得到了这个例外：

引起：java.lang.NoSuchMethodError：scala.reflect.api.JavaUniverse.runtimeMirror（Ljava / lang / ClassLoader;）Lscala / reflect / api / JavaMirrors $ JavaMirror; at org.elasticsearch.spark.serialization.ReflectionUtils $ .org $ elasticsearch $ spark $ serialization $ ReflectionUtils $$ checkCaseClass（ReflectionUtils.scala：42）在org.elasticsearch.spark.serialization.ReflectionUtils $$ anonfun $ checkCaseClassCache $ 1.apply（ReflectionUtils.scala：84） at org.elasticsearch.spark.serialization.ReflectionUtils $$ anonfun $ checkCaseClassCache $ 1.apply（ReflectionUtils.scala：83）

我知道我的问题是Scala版本（构建/运行）......

感谢您的帮助

EDIT BUILD POM：

  <build>
    <plugins>
        <plugin>
            <groupId>net.alchim31.maven</groupId>
            <artifactId>scala-maven-plugin</artifactId>
            <version>3.3.1</version>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>testCompile</goal>
                    </goals>
                    <configuration>
                        <args>
                            <argLine>-J-Xms128m</argLine>
                            <argLine>-J-Xmx512m</argLine>
                            <argLine>-J--XX:MaxPermSize=300m</argLine>
                            <argLine>-Djava.net.preferIPv4Stack=true</argLine>
                            <arg>-dependencyfile</arg>
                            <arg>${project.build.directory}/.scala_dependencies</arg>
                        </args>
                    </configuration>
                </execution>
            </executions>
        </plugin>


        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.3</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <shadedArtifactAttached>true</shadedArtifactAttached>
                        <shadedClassifierName>UBER</shadedClassifierName>
                        <artifactSet>
                            <includes>
                                <include>com.databricks:spark-csv_${scala.compact.version}</include>
                                <include>org.apache.commons:commons-csv</include>
                                <include>org.elasticsearch:elasticsearch-hadoop</include>
                            </includes>
                        </artifactSet>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Spark 2＆amp; Scala的Scala版本错误ElasticSearch 5.4.2

0 个答案: