Question

有没有办法让联合数据帧的数据帧在循环中？

这是一个示例代码

var fruits = List(
    "apple"
    ,"orange"
    ,"melon"

) 


for (x <- fruits){

           var df = Seq(("aaa","bbb",x)).toDF("aCol","bCol","name")

}

我想做点像

aCol | bCol | fruitsName
aaa,bbb,apple
aaa,bbb,orange
aaa,bbb,melon

再次感谢

Answer 1

Steffen Schmitz的回答是我认为最简洁的答案。如果您正在寻找更多自定义（字段类型等），则下面是更详细的答案：

import org.apache.spark.sql.types.{StructType, StructField, StringType}
import org.apache.spark.sql.Row

//initialize DF
val schema = StructType(
  StructField("aCol", StringType, true) ::
  StructField("bCol", StringType, true) ::
  StructField("name", StringType, true) :: Nil)
var initialDF = spark.createDataFrame(sc.emptyRDD[Row], schema)

//list to iterate through
var fruits = List(
    "apple"
    ,"orange"
    ,"melon"
)

for (x <- fruits) {
  //union returns a new dataset
  initialDF = initialDF.union(Seq(("aaa", "bbb", x)).toDF)
}

//initialDF.show()

的引用：

Answer 2

您可以创建一系列DataFrame s，然后使用reduce：

val results = fruits.
  map(fruit => Seq(("aaa", "bbb", fruit)).toDF("aCol","bCol","name")).
  reduce(_.union(_))

results.show()

Answer 3

在for循环中：

val fruits = List("apple", "orange", "melon")

( for(f <- fruits) yield ("aaa", "bbb", f) ).toDF("aCol", "bCol", "name")

Answer 4

如果您有不同/多个数据帧，则可以使用以下代码，这是有效的。

val newDFs = Seq(DF1,DF2,DF3)
newDFs.reduce(_ union _)

Answer 5

您可以先创建序列，然后使用toDF创建Dataframe。

scala> var dseq : Seq[(String,String,String)] = Seq[(String,String,String)]()
dseq: Seq[(String, String, String)] = List()

scala> for ( x <- fruits){
     |  dseq = dseq :+ ("aaa","bbb",x)
     | }

scala> dseq
res2: Seq[(String, String, String)] = List((aaa,bbb,apple), (aaa,bbb,orange), (aaa,bbb,melon))

scala> val df = dseq.toDF("aCol","bCol","name")
df: org.apache.spark.sql.DataFrame = [aCol: string, bCol: string, name: string]

scala> df.show
+----+----+------+
|aCol|bCol|  name|
+----+----+------+
| aaa| bbb| apple|
| aaa| bbb|orange|
| aaa| bbb| melon|
+----+----+------+

Answer 6

嗯......我认为你的问题有点误导。

根据我对你要做的事情的有限理解，你应该做以下事，

val fruits = List(
  "apple",
  "orange",
  "melon"
)

val df = fruits
  .map(x => ("aaa", "bbb", x))
  .toDF("aCol", "bCol", "name")

这应该足够了。

scala - Spark：如何在循环中结合所有数据帧

6 个答案: