我有CSV数据:
"id","price"
"1","79.07"
"2","91.27"
"3","85.6"
使用SparkSession
进行阅读:
def readToDs(resource: String, schema: StructType): Dataset = {
sparkSession.read
.option("header", "true")
.schema(schema)
.csv(resource)
.as[ItemPrice]
}
案例分类:
case class ItemPrice(id: Long, price: BigDecimal)
打印数据集:
def main(args: Array[String]): Unit = {
val prices: Dataset =
readToDs("src/main/resources/app/data.csv", Encoders.product[ItemPrice].schema);
prices.show();
}
输出:
+----------+--------------------+
| id| price|
+----------+--------------------+
| 1|79.07000000000000...|
| 2|91.27000000000000...|
| 3|85.60000000000000...|
+----------+--------------------+
所需的输出:
+----------+--------+
| id| price|
+----------+--------+
| 1| 79.07|
| 2| 91.27|
| 3| 85.6 |
+----------+--------+
我已经知道的选项:
使用硬编码的列顺序和数据类型手动定义架构,例如:
def defineSchema(): StructType =
StructType(
Seq(StructField("id", LongType, nullable = false)) :+
StructField("price", DecimalType(3, 2), nullable = false)
)
并像这样使用它:
val prices: Dataset = readToDs("src/main/resources/app/data.csv", defineSchema);
如何在不手动定义所有结构的情况下设置精度(3,2)
?
答案 0 :(得分:2)
假设您的csv为
scala> val df = Seq(("1","79.07","89.04"),("2","91.27","1.02"),("3","85.6","10.01")).toDF("item","price1","price2")
df: org.apache.spark.sql.DataFrame = [item: string, price1: string ... 1 more field]
scala> df.printSchema
root
|-- item: string (nullable = true)
|-- price1: string (nullable = true)
|-- price2: string (nullable = true)
您可以像下面一样投射
scala> val df2 = df.withColumn("price1",'price1.cast(DecimalType(4,2)))
df2: org.apache.spark.sql.DataFrame = [item: string, price1: decimal(4,2) ... 1 more field]
scala> df2.printSchema
root
|-- item: string (nullable = true)
|-- price1: decimal(4,2) (nullable = true)
|-- price2: string (nullable = true)
scala>
现在,如果您知道csv ..中带有数组的小数列列表,则可以像下面这样动态地进行
scala> import org.apache.spark.sql.types._
import org.apache.spark.sql.types._
scala> val decimal_cols = Array("price1","price2")
decimal_cols: Array[String] = Array(price1, price2)
scala> val df3 = decimal_cols.foldLeft(df){ (acc,r) => acc.withColumn(r,col(r).cast(DecimalType(4,2))) }
df3: org.apache.spark.sql.DataFrame = [item: string, price1: decimal(4,2) ... 1 more field]
scala> df3.show
+----+------+------+
|item|price1|price2|
+----+------+------+
| 1| 79.07| 89.04|
| 2| 91.27| 1.02|
| 3| 85.60| 10.01|
+----+------+------+
scala> df3.printSchema
root
|-- item: string (nullable = true)
|-- price1: decimal(4,2) (nullable = true)
|-- price2: decimal(4,2) (nullable = true)
scala>
有帮助吗?。
UPDATE1:
使用inferSchema读取csv文件,然后将所有双精度字段动态转换为DecimalType(4,2)。
val df = spark.read.format("csv").option("header","true").option("inferSchema","true").load("in/items.csv")
df.show
df.printSchema()
val decimal_cols = df.schema.filter( x=> x.dataType.toString == "DoubleType" ).map(x=>x.name)
// or df.schema.filter( x=> x.dataType==DoubleType )
val df3 = decimal_cols.foldLeft(df){ (acc,r) => acc.withColumn(r,col(r).cast(DecimalType(4,2))) }
df3.printSchema()
df3.show()
结果:
+-----+------+------+
|items|price1|price2|
+-----+------+------+
| 1| 79.07| 89.04|
| 2| 91.27| 1.02|
| 3| 85.6| 10.01|
+-----+------+------+
root
|-- items: integer (nullable = true)
|-- price1: double (nullable = true)
|-- price2: double (nullable = true)
root
|-- items: integer (nullable = true)
|-- price1: decimal(4,2) (nullable = true)
|-- price2: decimal(4,2) (nullable = true)
+-----+------+------+
|items|price1|price2|
+-----+------+------+
| 1| 79.07| 89.04|
| 2| 91.27| 1.02|
| 3| 85.60| 10.01|
+-----+------+------+
答案 1 :(得分:0)
一个选项是为输入模式定义转换器:
def defineDecimalType(schema: StructType): StructType = {
new StructType(
schema.map {
case StructField(name, dataType, nullable, metadata) =>
if (dataType.isInstanceOf[DecimalType])
// Pay attention to max precision in the source data
StructField(name, new DecimalType(20, 2), nullable, metadata)
else
StructField(name, dataType, nullable, metadata)
}.toArray
)
}
def main(args: Array[String]): Unit = {
val prices: Dataset =
readToDs("src/main/resources/app/data.csv", defineDecimalType(Encoders.product[ItemPrice].schema));
prices.show();
}
此方法的缺点是此映射应用于每个列,并且如果您的ID
不能满足精确度要求(假设ID = 10000
至DecimalType(3, 2)
),则您会得到一个例外:
原因:java.lang.IllegalArgumentException:要求失败: 十进制精度4超过最大精度3 scala.Predef $ .require(Predef.scala:224)在 org.apache.spark.sql.types.Decimal.set(Decimal.scala:113)在 org.apache.spark.sql.types.Decimal $ .apply(Decimal.scala:426)在 org.apache.spark.sql.execution.datasources.csv.CSVTypeCast $ .castTo(CSVInferSchema.scala:273) 在 org.apache.spark.sql.execution.datasources.csv.CSVRelation $$ anonfun $ csvParser $ 3.apply(CSVRelation.scala:125) 在 org.apache.spark.sql.execution.datasources.csv.CSVRelation $$ anonfun $ csvParser $ 3.apply(CSVRelation.scala:94) 在 org.apache.spark.sql.execution.datasources.csv.CSVFileFormat $$ anonfun $ buildReader $ 1 $$ anonfun $ apply $ 2.apply(CSVFileFormat.scala:167) 在 org.apache.spark.sql.execution.datasources.csv.CSVFileFormat $$ anonfun $ buildReader $ 1 $$ anonfun $ apply $ 2.apply(CSVFileFormat.scala:166)
这就是为什么保持精度高于源数据中最大小数的重要性:
if (dataType.isInstanceOf[DecimalType])
StructField(name, new DecimalType(20, 2), nullable, metadata)
答案 2 :(得分:-1)
我尝试使用2个不同的CSV文件加载示例数据,并且工作正常,结果与以下代码所预期的一样。 我在Windows上使用Spark 2.3.1。
//read with double quotes
val df1 = spark.read
.format("csv")
.option("header","true")
.option("inferSchema","true")
.option("nullValue","")
.option("mode","failfast")
.option("path","D:/bitbuket/spark-examples/53667822/string.csv")
.load()
df1.show
/*
scala> df1.show
+---+-----+
| id|price|
+---+-----+
| 1|79.07|
| 2|91.27|
| 3| 85.6|
+---+-----+
*/
//read with without quotes
val df2 = spark.read
.format("csv")
.option("header","true")
.option("inferSchema","true")
.option("nullValue","")
.option("mode","failfast")
.option("path","D:/bitbuket/spark-examples/53667822/int-double.csv")
.load()
df2.show
/*
scala> df2.show
+---+-----+
| id|price|
+---+-----+
| 1|79.07|
| 2|91.27|
| 3| 85.6|
+---+-----+
*/