我从Spark示例网站获得以下代码,尝试从Eclipse运行它,但似乎代码甚至无法编译。
import org.apache.spark._
import org.apache.spark.SparkContext._
object DataFrameExample {
def main(args: Array[String]) {
case class Person(name: String, age: Int)
val conf = new SparkConf().setAppName("wordCount"); //.setMaster("local")
conf.setMaster("local");
val sc = new SparkContext(conf)
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext._
import sqlContext.implicits._
val people = sc.textFile("examples/src/main/resources/people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt)).toDF()
people.registerTempTable("people")
val teenagers = sqlContext.sql("SELECT name, age FROM people WHERE age >= 13 AND age <= 19")
// The results of SQL queries are DataFrames and support all the normal RDD operations.
// The columns of a row in the result can be accessed by field index:
teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
// or by field name:
teenagers.map(t => "Name: " + t.getAs[String]("name")).collect().foreach(println)
// row.getValuesMap[T] retrieves multiple columns at once into a Map[String, T]
teenagers.map(_.getValuesMap[Any](List("name", "age"))).collect().foreach(println)
// Map("name" -> "Justin", "age" -> 19)
}
}
但后来我遇到了以下错误。我在这里错过了吗?谢谢!
同样的错误(来自IntelliJ的文本)
错误:(18,93)没有TypeTag可供Person使用 val people = sc.textFile(“examples / src / main / resources / people.txt”)。map(_。split(“,”))。map(p =&gt; Person(p(0),p(1 ).trim.toInt))。toDF() ^
答案 0 :(得分:2)
移动类Person
的定义:
case class Person(name: String, age: Int)
object DataFrameExample {
def main(args: Array[String]) {
// [...]
}
}
该定义必须在使用它的方法之外。
至于原因:看看this,从那里引用:
2-移动方法之外的案例类:
案例类,使用它来定义DataFrame的架构, 应该在需要它的方法之外定义。