Question

我有一个Spark Dataframe，每行都有一个评论。

+--------------------+
|          reviewText| 
+--------------------+
|Spiritually and m...|
|This is one my mu...|
|This book provide...|
|I first read THE ...|
+--------------------+

我试过了：

SplitSentences = df.withColumn("split_sent",sentencesplit_udf(col('reviewText')))
SplitSentences = SplitSentences.select(SplitSentences.split_sent)

然后我创建了函数：

def word_count(text):
    return len(text.split())

wordcount_udf = udf(lambda x: word_count(x))

df2 = SplitSentences.withColumn("word_count", 
  wordcount_udf(col('split_sent')).cast(IntegerType())

我想计算每个评论（行）中每个句子的单词，但它不起作用。

Answer 1

您可以使用split 内置函数来分割句子并使用size 内置函数将数组长度计为< / p>

df.withColumn("word_count", F.size(F.split(df['reviewText'], ' '))).show(truncate=False)

这样你就不会需要昂贵的udf功能

举个例子，假设你有一个句子数据框

+-----------------------------+
|reviewText                   |
+-----------------------------+
|this is text testing spliting|
+-----------------------------+

应用上述size和split功能后，您应该

+-----------------------------+----------+
|reviewText                   |word_count|
+-----------------------------+----------+
|this is text testing spliting|5         |
+-----------------------------+----------+

如果您在一行中有多个句子，如下所示

+----------------------------------------------------------------------------------+
|reviewText                                                                        |
+----------------------------------------------------------------------------------+
|this is text testing spliting. this is second sentence. And this is the third one.|
+----------------------------------------------------------------------------------+

然后你必须写一个udf函数，如下所示

from pyspark.sql import functions as F
def countWordsInEachSentences(array):
    return [len(x.split()) for x in array]

countWordsSentences = F.udf(lambda x: countWordsInEachSentences(x.split('. ')))

df.withColumn("word_count", countWordsSentences(df['reviewText'])).show(truncate=False)

应该给你

+----------------------------------------------------------------------------------+----------+
|reviewText                                                                        |word_count|
+----------------------------------------------------------------------------------+----------+
|this is text testing spliting. this is second sentence. And this is the third one.|[5, 4, 6] |
+----------------------------------------------------------------------------------+----------+

我希望答案很有帮助

计算每个句子Spark Dataframes中的单词数

1 个答案: