我正在使用Spark(2.0)开发Spark SQL并使用Java API读取CSV。
在CSV文件中有双引号,逗号分隔列。例如:"Express Air,Delivery Truck"
读取CSV并返回数据集的代码:
Dataset<Row> df = spark.read()
.format("com.databricks.spark.csv")
.option("inferSchema", "true")
.option("header", "true")
.load(filename)
结果:
+-----+--------------+--------------------------+
|Year | State | Ship Mode |...
+-----+--------------+--------------------------+
|2012 |New York |Express Air,Delivery Truck|...
|2013 |Nevada |Delivery Truck |...
|2013 |North Carolina|Regular Air,Delivery Truck|...
+-----+--------------+--------------------------+
但是,我想将Shop Mode
拆分为Mode1
和Mode2
列并作为数据集返回。
+-----+--------------+--------------+---------------+
|Year | State | Mode1 | Mode2 |...
+-----+--------------+--------------+---------------+
|2012 |New York |Express Air |Delivery Truck |...
|2013 |Nevada |Delivery Truck|null |...
|2013 |North Carolina|Regular Air |Delivery Truck |...
+-----+--------------+--------------+---------------+
有什么方法可以使用Java Spark做到这一点吗?
我尝试使用MapFunction,但call()方法没有返回Row。
Ship Mode
将为动态,即CSV可能包含一种或两种发货模式。
感谢。
答案 0 :(得分:2)
您可以使用 selectExpr ,这是一个接受 SQL表达式的select变体,如下所示:
df.selectExpr("Year","State","split(Ship Mode, ',')[0] as Mode1","split(Ship Mode, ',')[1] as Mode2");
结果是行数据集。
答案 1 :(得分:1)
我们可以:
例如:
import org.apache.spark.sql.functions._
import org.apache.spark.sql.{Column, Row}
val splitter = udf((str: String) => {
val splitted = str.split(",").lift
Array(splitted(0), splitted(1))
})
val dfShipMode = df.select($"year",$"state", splitter($"shipMode") as "modes")
.select($"year", $"state", $"modes"(0) as "mode1", $"modes"(1) as "mode2")