我在scala spark程序中遇到了过滤方法的编译时错误。下面是我的代码:
case class TrafficSchema(a: String, b: Int, c: Int, d: Int, e: Float)
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
val data = sqlContext.read.csv(input_Path)
val header = data.first()
val trackerData = data.filter(head => head != header)
.map(f => TrafficSchema(f.get(0).toString(), f.get(1).toString().toInt, f.get(2).toString().toInt, f.get(3).toString().toInt, f.get(4).toString().toFloat)).toDF()
我在过滤方法上遇到编译时错误,下面是错误,
Reference to method filter in class Dataset should not have survived past type checking, it should have been processed and eliminated during expansion of an enclosing macro.
以下是我的Maven依赖项:
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>2.0.0-cloudera1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.10</artifactId>
<version>2.0.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.5</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.25</version>
</dependency>
<dependency>
<groupId>org.clapper</groupId>
<artifactId>grizzled-slf4j_${scala.compat.version}</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>1.3.2</version>
</dependency>
<!-- Test Scopes -->
<dependency>
<groupId>org.scalactic</groupId>
<artifactId>scalactic_${scala.compat.version}</artifactId>
<version>3.0.5</version>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.compat.version}</artifactId>
<version>3.0.5</version>
<scope>test</scope>
</dependency>
</dependencies>
我正在使用的Scala版本是2.10.6 我不确定自己在做什么错。