Question

我有一个数据框，其列为“ EVENT_ID”，其数据类型为String。我正在运行FPGrowth算法，但抛出以下错误

Py4JJavaError: An error occurred while calling o1711.fit. 
:java.lang.IllegalArgumentException: requirement failed: 
The input column must be array, but got string.

EVENT_ID列具有值

E_34503_Probe
E_35203_In
E_31901_Cbc

我正在使用以下代码将字符串列转换为arraytype

df2 = df.withColumn("EVENT_ID", df["EVENT_ID"].cast(types.ArrayType(types.StringType())))

但是我收到以下错误

Py4JJavaError: An error occurred while calling o1874.withColumn.
: org.apache.spark.sql.AnalysisException: cannot resolve '`EVENT_ID`' due to data type mismatch: cannot cast string to array<string>;;

我如何将该列转换为数组类型或以字符串类型运行FPGrowth算法？

Answer 1

原始答案

尝试以下操作。

In  [0]: from pyspark.sql.types import StringType
         from pyspark.sql.functions import col, regexp_replace, split

In  [1]: df = spark.createDataFrame(["E_34503_Probe", "E_35203_In", "E_31901_Cbc"], StringType()).toDF("EVENT_ID")
         df.show()
Out [1]: +-------------+
         |     EVENT_ID|
         +-------------+
         |E_34503_Probe|
         |   E_35203_In|
         |  E_31901_Cbc|
         +-------------+

In  [2]: df_new = df.withColumn("EVENT_ID", split(regexp_replace(col("EVENT_ID"), r"(^\[)|(\]$)|(')", ""), ", "))
         df_new.printSchema()
Out [2]: root
          |-- EVENT_ID: array (nullable = true)
          |    |-- element: string (containsNull = true)

我希望这会有所帮助。

修改后的答案

正如@pault在他的comment中很好地指出的，以下是更简单的解决方案：

In  [0]: from pyspark.sql.types import StringType
         from pyspark.sql.functions import array

In  [1]: df = spark.createDataFrame(["E_34503_Probe", "E_35203_In", "E_31901_Cbc"], StringType()).toDF("EVENT_ID")
         df.show()
Out [1]: +-------------+
         |     EVENT_ID|
         +-------------+
         |E_34503_Probe|
         |   E_35203_In|
         |  E_31901_Cbc|
         +-------------+

In  [2]: df_new = df.withColumn("EVENT_ID", array(df["EVENT_ID"]))
         df_new.printSchema()
Out [2]: root
           |-- EVENT_ID: array (nullable = false)
           |    |-- element: string (containsNull = true)

在PySpark中将StringType列转换为ArrayType

1 个答案:

原始答案

修改后的答案