如何在spark DF中使用小写列值进行连接

Question

 val df_final = Data1
        .join(broadcast(df),
          Seq(lower(col("NAME")),  lower(col("TYPE"))),
          "left")

上面的代码抛出了以下错误。

<console>:65: error: overloaded method value join with alternatives:
  (right: org.apache.spark.sql.Dataset[_],joinExprs: org.apache.spark.sql.Column,joinType: String)org.apache.spark.sql.DataFrame <and>
  (right: org.apache.spark.sql.Dataset[_],usingColumns: Seq[String],joinType: String)org.apache.spark.sql.DataFrame
 cannot be applied to (org.apache.spark.sql.Dataset[org.apache.spark.sql.Row], Seq[org.apache.spark.sql.Column], String)
               .join(broadcast(df),
            ^

Answer 1

您无法在条件序列中使用内置函数，而是可以执行以下操作

    @JsonSerialize(using = AlternativeQuantitySerializer.class)
    private Boundary<Quantity> to;