Question

当列名为空值时，我没有任何想法来获取列名

例如，

case class A(name: String, id: String, email: String, company: String)

val e1 = A("n1", null, "n1@c1.com", null)
val e2 = A("n2", null, "n2@c1.com", null)
val e3 = A("n3", null, "n3@c1.com", null)
val e4 = A("n4", null, "n4@c2.com", null)
val e5 = A("n5", null, "n5@c2.com", null)
val e6 = A("n6", null, "n6@c2.com", null)
val e7 = A("n7", null, "n7@c3.com", null)
val e8 = A("n8", null, "n8@c3.com", null)
val As = Seq(e1, e2, e3, e4, e5, e6, e7, e8)
val df = sc.parallelize(As).toDF

此代码使数据框如下：

+----+----+---------+-------+
|name|  id|    email|company|
+----+----+---------+-------+
|  n1|null|n1@c1.com|   null|
|  n2|null|n2@c1.com|   null|
|  n3|null|n3@c1.com|   null|
|  n4|null|n4@c2.com|   null|
|  n5|null|n5@c2.com|   null|
|  n6|null|n6@c2.com|   null|
|  n7|null|n7@c3.com|   null|
|  n8|null|n8@c3.com|   null|
+----+----+---------+-------+

我希望列名称的所有行都为null：id，company

我不关心输出的类型。数组，字符串，RDD无论

Answer 1

您可以对所有列进行简单计数，然后使用返回计数0的列的索引，将子集df.columns：

import org.apache.spark.sql.functions.{count,col}
// Get column indices
val col_inds = df.select(df.columns.map(c => count(col(c)).alias(c)): _*)
                 .collect()(0)
                 .toSeq.zipWithIndex
                 .filter(_._1 == 0).map(_._2)
// Subset column names using the indices
col_inds.map(i => df.columns.apply(i))
//Seq[String] = ArrayBuffer(id, company)

Answer 2

另一种解决方案可能如下（但恐怕性能可能不令人满意）。

val ids = Seq(
  ("1", null: String), 
  ("1", null: String),
  ("10", null: String)
).toDF("id", "all_nulls")

scala> ids.show
+---+---------+
| id|all_nulls|
+---+---------+
|  1|     null|
|  1|     null|
| 10|     null|
+---+---------+

val s = ids.columns.
  map { c => 
    (c, ids.select(c).dropDuplicates(c).na.drop.count) }. // <-- performance here!
  collect { case (c, cnt) if cnt == 0 => c }
scala> s.foreach(println)
all_nulls

如何获取所有值为null的列名？

2 个答案: