Question

我有以下代码：

from pyspark.sql.types import *
from pyspark.ml.linalg import Vectors, VectorUDT
import pyspark.sql.functions as F

dot_udf = F.udf(lambda x,y: float(x.dot(y)), DoubleType())
l = [(Vectors.dense([1, 2, 3, 4 ,5]), Vectors.dense([5, 4, 3, 2, 1]),),
     (Vectors.dense([0, 4, 8, 2, 1]), None,),
     (None, Vectors.dense([5, 0, 3, 9, 1]),),
     ]

def finish(row):
    new_row = []
    new_row.append(None if row['my_row_1'] == None else Vectors.dense(row['my_row_1']))
    new_row.append(None if row['my_row_2'] == None else Vectors.dense(row['my_row_2']))
    return new_row


with (SparkSession
      .builder
      .appName('test_mtassoni')
      .getOrCreate()) as spark:
    schema = StructType([StructField('my_row_1', VectorUDT(), True),
                         StructField('my_row_2', VectorUDT(), True)])

    df = spark.createDataFrame(l, schema)
    rdd = df.rdd
    rdd = rdd.map(finish)

    out_schema = StructType([StructField('my_row_1', VectorUDT(), True),
                             StructField('my_row_2', VectorUDT(), True)])
    fdf = spark.createDataFrame(rdd, schema=out_schema)

    fdf = fdf.withColumn('row_sim', F.when(((F.col('my_row_1').isNull()) |
                                            (F.col('my_row_2').isNull())),
                                       np.nan).otherwise(dot_udf(fdf.my_row_1, fdf.my_row_2))
                     )
    fdf.show()

它在最后一个命令上失败，并显示以下TypeError：

TypeError: Cannot treat type <type 'NoneType'> as a vector

有人知道如何解决吗？提前非常感谢您。

Answer 1

您的方法存在的问题是您在udf部分调用了otherwise函数，但事实是每一行都被传递给udf函数。因此，问题在于，您正在将按列函数（when/otherwise）与按行函数（udf）组合

解决方案是将when/otherwise函数内的udf部分移动为

def dotProduct(x, y):
    if(x == None or y == None):
        return np.nan
    else:
        return float(x.dot(y))

dot_udf = F.udf(lambda x,y: dotProduct(x, y), DoubleType())

，然后分别以

调用udf函数

fdf = fdf.withColumn('row_sim', dot_udf(fdf.my_row_1, fdf.my_row_2))
fdf.show()

应该为您提供正确的结果，而不会出现

错误

+--------------------+--------------------+-------+
|            my_row_1|            my_row_2|row_sim|
+--------------------+--------------------+-------+
|[1.0,2.0,3.0,4.0,...|[5.0,4.0,3.0,2.0,...|   35.0|
|[0.0,4.0,8.0,2.0,...|                null|    NaN|
|                null|[5.0,0.0,3.0,9.0,...|    NaN|
+--------------------+--------------------+-------+

我希望答案会有所帮助

Pyspark错误：TypeError：无法将类型<type'nonetype'=“”>视为向量

1 个答案: