在Scala Spark程序的过滤器方法中获取编译时错误

时间:2018-12-05 06:20:52

标签: scala apache-spark

我在scala spark程序中遇到了过滤方法的编译时错误。下面是我的代码:

case class TrafficSchema(a: String, b: Int, c: Int, d: Int, e: Float)

val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
val data = sqlContext.read.csv(input_Path)

val header = data.first()
val trackerData = data.filter(head => head != header)
  .map(f => TrafficSchema(f.get(0).toString(), f.get(1).toString().toInt, f.get(2).toString().toInt, f.get(3).toString().toInt, f.get(4).toString().toFloat)).toDF()

我在过滤方法上遇到编译时错误,下面是错误,

Reference to method filter in class Dataset should not have survived past type checking, it should have been processed and eliminated during expansion of an enclosing macro.

以下是我的Maven依赖项:

    <dependencies>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>2.0.0-cloudera1-SNAPSHOT</version>
        </dependency>
        <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.10</artifactId>
         <version>2.0.0</version>
    </dependency>
    <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.10</artifactId>
            <version>2.0.0</version>
            <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.5</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.7.25</version>
    </dependency>
    <dependency>
        <groupId>org.clapper</groupId>
        <artifactId>grizzled-slf4j_${scala.compat.version}</artifactId>
        <version>1.3.1</version>
    </dependency>
    <dependency>
        <groupId>com.typesafe</groupId>
        <artifactId>config</artifactId>
        <version>1.3.2</version>
    </dependency>
    <!-- Test Scopes -->
    <dependency>
        <groupId>org.scalactic</groupId>
        <artifactId>scalactic_${scala.compat.version}</artifactId>
        <version>3.0.5</version>
    </dependency>
    <dependency>
        <groupId>org.scalatest</groupId>
        <artifactId>scalatest_${scala.compat.version}</artifactId>
        <version>3.0.5</version>
        <scope>test</scope>
    </dependency>
</dependencies>

我正在使用的Scala版本是2.10.6 我不确定自己在做什么错。

0 个答案:

没有答案