我在Java中使用spark-sql-2.4.3v。 我有下面的情况
+---+-----+-------+------+-----+
| id| code|entity|value1|value2|
+---+-----+-------+------+-----+
| 20|score|school| a| aaa|
| 21|score|school| aa| aa|
| 22| rate|school| 11| 14|
| 23|score|school| aaaa| a|
| 24| rate|school| 12| 12|
| 25|score|school| aaaa| a|
+---+-----+------+------+------+
我只需要对最终输出中的“代码”与“值”进行映射,仅适用于在“数据”数据框中具有“代码”作为“得分”的那些行/记录。
如何从codeValudeDf中查找哈希图,以便获得如下输出
{{1}}
是否有可能使此查询最佳化,即每次我都不应该从DB中提取数据帧数据时?
答案 0 :(得分:2)
如果查找数据的大小较小,则可以创建Map
和broadcast
。 broadcasted map
可以在udf中轻松使用,如下所示-
val data = List(
("20", "score", "school", 14 ,12),
("21", "score", "school", 13 , 13),
("22", "rate", "school", 11 ,14),
("23", "score", "school", 11 ,14),
("24", "rate", "school", 12 ,12),
("25", "score", "school", 11 ,14)
)
val df = data.toDF("id", "code", "entity", "value1","value2")
df.show
/**
* +---+-----+------+------+------+
* | id| code|entity|value1|value2|
* +---+-----+------+------+------+
* | 20|score|school| 14| 12|
* | 21|score|school| 13| 13|
* | 22| rate|school| 11| 14|
* | 23|score|school| 11| 14|
* | 24| rate|school| 12| 12|
* | 25|score|school| 11| 14|
* +---+-----+------+------+------+
*/
//this look up data populated from DB.
val ll = List(
("aaaa", 11),
("aaa", 12),
("aa", 13),
("a", 14)
)
val codeValudeDf = ll.toDF( "code", "value")
codeValudeDf.show
/**
* +----+-----+
* |code|value|
* +----+-----+
* |aaaa| 11|
* | aaa| 12|
* | aa| 13|
* | a| 14|
* +----+-----+
*/
broadcasted map
可以在udf中轻松使用,如下所示-
val lookUp = spark.sparkContext
.broadcast(codeValudeDf.map{case Row(code: String, value: Integer) => value -> code}
.collect().toMap)
val look_up = udf((value: Integer) => lookUp.value.get(value))
df.withColumn("value1",
when($"code" === "score", look_up($"value1")).otherwise($"value1".cast("string")))
.withColumn("value2",
when($"code" === "score", look_up($"value2")).otherwise($"value2".cast("string")))
.show(false)
/**
* +---+-----+------+------+------+
* |id |code |entity|value1|value2|
* +---+-----+------+------+------+
* |20 |score|school|a |aaa |
* |21 |score|school|aa |aa |
* |22 |rate |school|11 |14 |
* |23 |score|school|aaaa |a |
* |24 |rate |school|12 |12 |
* |25 |score|school|aaaa |a |
* +---+-----+------+------+------+
*/
答案 1 :(得分:-2)
使用广播的地图确实是一个明智的决定,因为您无需每次都访问数据库即可提取查找数据。
在这里,我已经使用UDF中的键值映射解决了该问题。我无法比较它的性能广播地图方法,但欢迎火花专家发表意见。
步骤#1:构建KeyValueMap-
val data = List(
("20", "score", "school", 14 ,12),
("21", "score", "school", 13 , 13),
("22", "rate", "school", 11 ,14),
("23", "score", "school", 11 ,14),
("24", "rate", "school", 12 ,12),
("25", "score", "school", 11 ,14)
)
val df = data.toDF("id", "code", "entity", "value1","value2")
val ll = List(
("aaaa", 11),
("aaa", 12),
("aa", 13),
("a", 14)
)
val codeValudeDf = ll.toDF( "code", "value")
val Keys = codeValudeDf.select("value").collect().map(_(0).toString).toList
val Values = codeValudeDf.select("code").collect().map(_(0).toString).toList
val KeyValueMap = Keys.zip(Values).toMap
步骤#2:创建UDF
def CodeToValue(code: String, key: String): String = {
if (key == null) return ""
if (code != "score") return key
val result: String = KeyValueMap.getOrElse(key,"not found!")
return result }
val CodeToValueUDF = udf (CodeToValue(_:String, _:String):String )
第3步:在原始数据框中使用UDF添加派生列
val newdf = df.withColumn("Col1", CodeToValueUDF(col("code"), col("value1")))
val finaldf = newdf.withColumn("Col2", CodeToValueUDF(col("code"), col("value2")))
finaldf.show(false)
+---+-----+------+------+------+----+----+
| id| code|entity|value1|value2|Col1|Col2|
+---+-----+------+------+------+----+----+
| 20|score|school| 14| 12| a| aaa|
| 21|score|school| 13| 13| aa| aa|
| 22| rate|school| 11| 14| 11| 14|
| 23|score|school| 11| 14|aaaa| a|
| 24| rate|school| 12| 12| 12| 12|
| 25|score|school| 11| 14|aaaa| a|
+---+-----+------+------+------+----+----+