Question

我想创建并保存一个充满随机int的表格。到目前为止，一切都很顺利，但我不明白我如何能够将多维数组tmp放入Dataframe，并在顶部定义架构。

import org.apache.spark.sql.types.{
StructType, StructField, StringType, IntegerType, DoubleType}
import org.apache.spark.sql.Row

val schema = StructType(
StructField("rowId", IntegerType, true) ::
StructField("t0_1", DoubleType, true) ::
StructField("t0_2", DoubleType, true) ::    
StructField("t0_3", DoubleType, true) ::
StructField("t0_4", DoubleType, true) ::
StructField("t0_5", DoubleType, true) ::
StructField("t0_6", DoubleType, true) ::
StructField("t0_7", DoubleType, true) ::
StructField("t0_8", DoubleType, true) ::
StructField("t0_9", DoubleType, true) ::
StructField("t0_10", DoubleType, true) :: Nil)

val columnNo = 10;
val rowNo = 50;

var c = 0;
var r = 0;

val tmp = Array.ofDim[Double](10,rowNo)

for (r <- 1 to rowNo){
for (c <- 1 to columnNo){
    val temp = new scala.util.Random
    tmp(c-1)(r-1) = temp.nextDouble
    println( "Value of " + c + "/"+ r + ":" + tmp(c-1)(r-1));
}
}

val df = sc.parallelize(tmp).toDF
df.show
dataframe.show

Answer 1

您无法将数组数组转换为DataFrame，而是需要一个元组数组的案例类。这里的变体基于与您想要的模式对应的案例类：

case class Record(
  rowID:Option[Int],
  t0_1:Option[Double],
  t0_2:Option[Double],
  t0_3:Option[Double],
  t0_4:Option[Double],
  t0_5:Option[Double],
  t0_6:Option[Double],
  t0_7:Option[Double],
  t0_8:Option[Double],
  t0_9:Option[Double],
  t0_10:Option[Double]
)

val rowNo = 50;
val temp = new scala.util.Random

val data = (1 to rowNo).map(r => 
 Record(
    Some(r),
    Some(temp.nextDouble),
    Some(temp.nextDouble),
    Some(temp.nextDouble),
    Some(temp.nextDouble),
    Some(temp.nextDouble),
    Some(temp.nextDouble),
    Some(temp.nextDouble),
    Some(temp.nextDouble),
    Some(temp.nextDouble),
    Some(temp.nextDouble)
  )
)

val df = sc.parallelize(data).toDF

Scala - 将数组数据转换为表或数据帧？

1 个答案: