Question

当我们使用

时，我对这种差异感到困惑

 df.filter(col("c1") === null) and df.filter(col("c1").isNull)

相同的数据帧我得到了重视 === null但isNull为零。请帮我理解其中的区别。谢谢

Answer 1

首先，除非出于兼容性原因，否则不要在Scala代码中使用null。

关于你的问题，这是一个简单的SQL。 col("c1") === null被解释为c1 = NULL，并且由于NULL标记了未定义的值，因此对于包括NULL本身在内的任何值都未定义结果。

spark.sql("SELECT NULL = NULL").show

+-------------+
|(NULL = NULL)|
+-------------+
|         null|
+-------------+

spark.sql("SELECT NULL != NULL").show

+-------------------+
|(NOT (NULL = NULL))|
+-------------------+
|               null|
+-------------------+

spark.sql("SELECT TRUE != NULL").show

+------------------------------------+
|(NOT (true = CAST(NULL AS BOOLEAN)))|
+------------------------------------+
|                                null|
+------------------------------------+

spark.sql("SELECT TRUE = NULL").show

+------------------------------+
|(true = CAST(NULL AS BOOLEAN))|
+------------------------------+
|                          null|
+------------------------------+

检查NULL的唯一有效方法是：

IS NULL：

spark.sql("SELECT NULL IS NULL").show

+--------------+
|(NULL IS NULL)|
+--------------+
|          true|
+--------------+

spark.sql("SELECT TRUE IS NULL").show

+--------------+
|(true IS NULL)|
+--------------+
|         false|
+--------------+

IS NOT NULL：

spark.sql("SELECT NULL IS NOT NULL").show

+------------------+
|(NULL IS NOT NULL)|
+------------------+
|             false|
+------------------+

spark.sql("SELECT TRUE IS NOT NULL").show

+------------------+
|(true IS NOT NULL)|
+------------------+
|              true|
+------------------+

分别在DataFrame DSL中实现Column.isNull和Column.isNotNull。

注意：

对于NULL - 安全比较使用IS DISTINCT / IS NOT DISTINCT：

spark.sql("SELECT NULL IS NOT DISTINCT FROM NULL").show

+---------------+ |(NULL <=> NULL)| +---------------+ | true| +---------------+

spark.sql("SELECT NULL IS NOT DISTINCT FROM TRUE").show

+--------------------------------+ |(CAST(NULL AS BOOLEAN) <=> true)| +--------------------------------+ | false| +--------------------------------+

或not(_ <=> _) / <=>

spark.sql("SELECT NULL AS col1, NULL AS col2").select($"col1" <=> $"col2").show

+---------------+ |(col1 <=> col2)| +---------------+ | true| +---------------+

spark.sql("SELECT NULL AS col1, TRUE AS col2").select($"col1" <=> $"col2").show

+---------------+ |(col1 <=> col2)| +---------------+ | false| +---------------+

分别在SQL和DataFrame DSL中。

相关：

Including null values in an Apache Spark Join

Answer 2

通常，解决Spark Dataframes中意外结果的最佳方法是查看解释计划。请考虑以下示例：

import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.functions._

object Example extends App {

  val session = SparkSession.builder().master("local[*]").getOrCreate()
  case class Record(c1: String, c2: String)
  val data = List(Record("a", "b"), Record(null, "c"))
  val rdd = session.sparkContext.parallelize(data)
  import session.implicits._

  val df: DataFrame = rdd.toDF
  val filtered = df.filter(col("c1") === null)
  println(filtered.count()) // <-- outputs 0, not expected

  val filtered2 = df.filter(col("c1").isNull)
  println(filtered2.count())
  println(filtered2) // <- outputs 1, as expected

  filtered.explain(true)
  filtered2.explain(true)
}

第一个解释计划显示：

== Physical Plan ==
*Filter (isnotnull(c1#2) && null)
+- Scan ExistingRDD[c1#2,c2#3]
== Parsed Logical Plan ==
'Filter isnull('c1)
+- LogicalRDD [c1#2, c2#3]

此过滤条款看似荒谬。 &&到null可确保永远无法解析为true。

第二个解释计划如下：

== Physical Plan ==
*Filter isnull(c1#2)
+- Scan ExistingRDD[c1#2,c2#3]

这里的过滤器是期望和想要的。

Spark DataDrame中=== null和isNull之间的差异

2 个答案: