我试图创建一个简单的DataFrame
,如下所示:
import sqlContext.implicits._
val lookup = Array("one", "two", "three", "four", "five")
val theRow = Array("1",Array(1,2,3), Array(0.1,0.4,0.5))
val theRdd = sc.makeRDD(theRow)
case class X(id: String, indices: Array[Integer], weights: Array[Float] )
val df = theRdd.map{
case Array(s0,s1,s2) => X(s0.asInstanceOf[String],s1.asInstanceOf[Array[Integer]],s2.asInstanceOf[Array[Float]])
}.toDF()
df.show()
df定义为
df: org.apache.spark.sql.DataFrame = [id: string, indices: array<int>, weights: array<float>]
这就是我想要的。
执行后,我
org.apache.spark.SparkException:作业因阶段失败而中止:阶段13.0中的任务1失败1次,最近失败:阶段13.0中失去的任务1.0(TID 50,localhost):scala.MatchError:1(类java.lang.String)
这个MatchError来自哪里?并且,是否有更简单的方法以编程方式创建样本DataFrames
?
答案 0 :(得分:2)
首先,theRow
应该是Row
而不是Array
。现在,如果您以尊重Java和Scala之间的兼容性的方式修改类型,那么您的示例将起作用
val theRow =Row("1",Array[java.lang.Integer](1,2,3), Array[Double](0.1,0.4,0.5))
val theRdd = sc.makeRDD(Array(theRow))
case class X(id: String, indices: Array[Integer], weights: Array[Double] )
val df=theRdd.map{
case Row(s0,s1,s2)=>X(s0.asInstanceOf[String],s1.asInstanceOf[Array[Integer]],s2.asInstanceOf[Array[Double]])
}.toDF()
df.show()
//+---+---------+---------------+
//| id| indices| weights|
//+---+---------+---------------+
//| 1|[1, 2, 3]|[0.1, 0.4, 0.5]|
//+---+---------+---------------+
答案 1 :(得分:0)
有关其他示例,您可以参考
import spark.implicits._
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val columns=Array("id", "first", "last", "year")
val df1=sc.parallelize(Seq(
(1, "John", "Doe", 1986),
(2, "Ive", "Fish", 1990),
(4, "John", "Wayne", 1995)
)).toDF(columns: _*)
val df2=sc.parallelize(Seq(
(1, "John", "Doe", 1986),
(2, "IveNew", "Fish", 1990),
(3, "San", "Simon", 1974)
)).toDF(columns: _*)