变量声明出错 - java.lang.NoSuchMethodError：scala.runtime.IntRef.create（I）Lscala / runtime / IntRef; - scala版本不匹配

时间：2017-10-26 15:45:36

标签： scala maven apache-spark intellij-idea

我在使用Maven的intelliJ上使用Spark / Scala应用程序。我正在尝试编写一个文件，在写完成后，我试图逐行读取它并根据条件递增一个名为count的变量。这是我的代码：

val writer: BufferedWriter = new BufferedWriter(new FileWriter("test.csv"))
/* Logic
    for
   writing 
  the file
*/
writer.close()

var count = 0
val bufferedSource = Source.fromFile("test.csv")

for(line <- bufferedSource.getLines()){
    count = count+1
    line.split(",").foreach(x => {if(x.toString == "1") count = count+1})
}

bufferedSource.close()

当我运行上面的代码时，它会给我以下错误：

Exception in thread "main" java.lang.NoSuchMethodError: scala.runtime.IntRef.create(I)Lscala/runtime/IntRef;

它在变量声明语句中抛出错误：var count = 0

我在网上搜索了这个错误，并找到了scala version mismatch的原因，如下所示：NoSuchMethodError when declaring a variable

最初，我将项目SDK设置为2.10.6。这是我用于项目的默认版本。我没有在我的pom.xml中提到任何scala语言依赖。当我运行mvn -U package命令来构建jar时，我收到以下消息：

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building project
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ IPscan ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\zkh9xcl\workspace\IPscan\src\main\resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ IPscan ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.7
[WARNING]  com.databricks:spark-csv_2.11:1.4.0 requires scala version: 2.11.7
[WARNING]  com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING] Multiple versions of scala libraries detected!
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ IPscan ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ IPscan ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory C:\Users\zkh9xcl\workspace\IPscan\src\test\resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile) @ IPscan ---
[WARNING]  Expected all dependencies to require Scala version: 2.11.7
[WARNING]  com.databricks:spark-csv_2.11:1.4.0 requires scala version: 2.11.7
[WARNING]  com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ IPscan ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ IPscan ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ IPscan ---
[INFO]
[INFO] --- maven-shade-plugin:2.4.1:shade (default) @ IPscan ---
[INFO] Including com.databricks:spark-csv_2.11:jar:1.4.0 in the shaded jar.
[INFO] Including org.scala-lang:scala-library:jar:2.11.7 in the shaded jar.
[INFO] Including org.apache.commons:commons-csv:jar:1.1 in the shaded jar.
[INFO] Including com.univocity:univocity-parsers:jar:1.5.1 in the shaded jar.
[INFO] Skipping pom dependency org.jodd:jodd:pom:3.4.0 in the shaded jar.
[INFO] Including org.threeten:threetenbp:jar:1.3.3 in the shaded jar.
[INFO] Attaching shaded artifact.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.054 s
[INFO] Finished at: 2017-10-26T11:18:39-04:00
[INFO] Final Memory: 38M/663M
[INFO] ------------------------------------------------------------------------

警告说：[WARNING] Multiple versions of scala libraries detected!

这是我的pom.xml：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>project</groupId>
  <artifactId></artifactId>
  <version>1.0-SNAPSHOT</version>
  <name>IPscan</name>
  <repositories>
    <repository>
      <id>scala-tools.org</id>
      <name>Scala-tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <id>scala-tools.org</id>
      <name>Scala-tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </pluginRepository>
  </pluginRepositories>
  <build>
    <sourceDirectory>src</sourceDirectory>
    <plugins>

      <plugin>
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.2.2</version>
        <executions>
          <execution>
            <id>scala-compile-first</id>
            <phase>process-resources</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>

          <execution>
            <id>scala-test-compile</id>
            <phase>process-test-resources</phase>
            <goals>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.1</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <relocations>
                <relocation>
                  <pattern>org.apache.http</pattern>
                  <shadedPattern>org.shaded.apache.http</shadedPattern>
                </relocation>
              </relocations>
              <filters>
                <filter>
                  <artifact>*:*</artifact>

                </filter>
              </filters>
              <shadedArtifactAttached>true</shadedArtifactAttached>
              <shadedClassifierName>shaded</shadedClassifierName>
            </configuration>

          </execution>
        </executions>
      </plugin>
    </plugins>   
  </build>    
  <dependencies>
    <dependency>
      <groupId>com.databricks</groupId>
      <artifactId>spark-csv_2.11</artifactId>
      <version>1.4.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.jodd/jodd -->
    <dependency>
      <groupId>org.jodd</groupId>
      <artifactId>jodd</artifactId>
      <version>3.4.0</version>
      <type>pom</type>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10 -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.10</artifactId>
      <version>1.6.0</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-api</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>jcl-over-slf4j</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.threeten/threetenbp -->
    <dependency>
      <groupId>org.threeten</groupId>
      <artifactId>threetenbp</artifactId>
      <version>1.3.3</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive_2.10 -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.10</artifactId>
      <version>1.6.0</version>
      <scope>provided</scope>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-api</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
  </dependencies>
</project>

我是maven的新手，如果我的pom.xml有疑问，请原谅。

我不确定为什么jar中包含多个版本的scala库。我实际上将java版本更改为2.11.7，然后更改为IntelliJ中项目的全局库中的2.10.4，但它提出了Library scala-sdk-2.10.4 is not used [Fix]的问题。我很确定我已将2.10.4添加到intelliJ中的依赖项列表中。无论我做什么，当我构建jar并且multiple scala versions仍然存在时，我仍然会收到noSuchMethodError警告。看起来我错过了什么。

导致错误的原因是什么？什么是克服这一点并在将来避免它的最佳方法？任何帮助，将不胜感激。谢谢！

P.S：我使用的是Spark 1.6，我无法升级到更新的版本。

1 个答案:

答案 0 :(得分：0)

它是依赖性不匹配的简单情况。我的猜测是因为spark-csv包裹com.databricks。您可以将版本更改为2.10，然后重试。

如果这有帮助，请告诉我。欢呼声。

相关问题
最新问题