为每个数据框行创建一个CSV文件

Question

我需要为每个数据框行创建一个数据框。

以下代码将使用数据框信息创建单个csv

import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.SparkConf
import java.sql.Timestamp
import org.apache.spark.sql._
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType, LongType, DoubleType};
import org.apache.spark.sql.functions._

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
var myDF = sqlContext.sql("select a, b, c from my_table")

val filename = "/tmp/myCSV.csv";
myDF.repartition(1).write.option("header", "true").option("compression", "none").option("timestampFormat", "yyyy-MM-dd'T'HH:mm:ss.SSS").csv(filename)

我想为每一行创建一个CSV

Answer 1

scala> val myDF = sqlContext.sql("select a, b, c from my_table")

scala> val c = myDF.cache.count //Let say total 100 records

scala> val newDF = myDF.repartition(c.toInt)
scala> newDF.rdd.getNumPartitions
res34: Int = 100

scala> newDF.write.format("csv").option("header","true").save(<path to write>)