看起来spark sql对“喜欢”查询区分大小写,对吗?
spark.sql("select distinct status, length(status) from table")
返回
Active|6
spark.sql("select distinct status from table where status like '%active%'")
不返回任何值
spark.sql("select distinct status from table where status like '%Active%'")
返回
Active
答案 0 :(得分:4)
是的,Spark区分大小写。默认情况下,大多数RDBMS区分大小写以进行字符串比较。如果您希望不区分大小写,请尝试rlike或将列转换为大写/小写。
scala> val df = Seq(("Active"),("Stable"),("Inactive")).toDF("status")
df: org.apache.spark.sql.DataFrame = [status: string]
scala> df.createOrReplaceTempView("tbl")
scala> df.show
+--------+
| status|
+--------+
| Active|
| Stable|
|Inactive|
+--------+
scala> spark.sql(""" select status from tbl where status like '%Active%' """).show
+------+
|status|
+------+
|Active|
+------+
scala> spark.sql(""" select status from tbl where lower(status) like '%active%' """).show
+--------+
| status|
+--------+
| Active|
|Inactive|
+--------+
scala>