pyspark中的案例声明问题

Question

我正在使用下面的代码来获取一个数据帧，但出现错误。如果将从代码中删除case语句，那么它将正常工作。

from pyspark.sql import functions as F
ivr_icm_cfs_lookup = spark.table("sample_table").where(F.col("tier") == "Servicing").select("fig","fig_bu","tier") .withColumn("fig",F.regexp_replace(F.col("fig"),"[^0-9a-zA-Z]",' ')).withColumn("fig_bu",F.when((F.col("fig_bu") == 'OPEN' & F.col("tier") == 'Academy', F.upper("tier")).otherwise(F.col("fig_bu"))))

Answer 1

先生，您的方括号有问题。立即尝试

from pyspark.sql import functions as F
ivr_icm_cfs_lookup = spark.table("sample_table").where(F.col("tier") == "Servicing").select("fig","fig_bu","tier") .withColumn("fig",F.regexp_replace(F.col("fig"),"[^0-9a-zA-Z]",' ')).withColumn("fig_bu",F.when((F.col("fig_bu") == 'OPEN') & (F.col("tier") == 'Academy'), F.upper("tier")).otherwise(F.col("fig_bu")))

如果这确实不适合您，则可以通过将数据帧注册为表然后在sql中编写案例来尝试使用spark sql。

from pyspark.sql import functions as F
ivr_icm_cfs_lookup = spark.table("sample_table").where(F.col("tier") == "Servicing").select("fig","fig_bu","tier") .withColumn("fig",F.regexp_replace(F.col("fig"),"[^0-9a-zA-Z]",' '))

ivr_icm_cfs_lookup.createOrReplaceTempView('ivr_icm_cfs_lookup_table')
ivr_icm_cfs_lookup_df = spark.sql('''
select *, case when fig_bu='OPEN' and tier='Academy' then upper(tier) else fig_bu END as fig_bu_2 from ivr_icm_cfs_lookup_table
'''

这样，您可以添加所需的列。