Question

编辑：答案：这是一个创建冲突的JAR文件！ 相关帖子为：Must include log4J, but it is causing errors in Apache Spark shell. How to avoid errors?

执行以下操作：

val numOfProcessors:Int = 2
val filePath:java.lang.String = "s3n://somefile.csv"
var rdd:org.apache.spark.rdd.RDD[java.lang.String] = sc.textFile(filePath, numOfProcessors)

我得到了

    error: type mismatch;
 found   : org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
 required: org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
       var rdd:org.apache.spark.rdd.RDD[java.lang.String] = sc.textFile(filePath, numOfProcessors)

编辑：第二个案例

val numOfProcessors = 2
val filePath = "s3n://somefile.csv"
var rdd = sc.textFile(filePath, numOfProcessors) //OK!

def doStuff(rdd: RDD[String]): RDD[String] = {rdd}

doStuff(rdd)

我明白了：

error: type mismatch;
 found   : org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
 required: org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.org.apache.spark.rdd.RDD[String]
              doStuff(rdd)
                      ^

没有评论......

为什么我收到此错误的任何想法？

Answer 1

问题是一个产生冲突的JAR文件。

Apache Spark类型不匹配的相同类型（String）

1 个答案: