Question

我被赋予了在Python Spark中创建单词计数程序的任务。我应该计算每个字母开头的单词数。

这是我编写的代码，但似乎无法获得结果。谁能帮助我进行故障排除？

in.txt内容：人们并不像他们看起来的那么美丽，当他们走路或说话时。他们只有自己喜欢的美丽就像他们分享一样。

import re
import sys
from pyspark import SparkConf, SparkContext

conf = SparkConf()
sc = SparkContext(conf=conf)
inRDD = sc.textFile("in.txt")
words = inRDD.flatMap(lambda l: re.split(" ",l))
LetterCount = words.map(lambda s : (s[0],1))
result = LetterCount.reduceByKey(lambda n1, n2 : n1 + n2)

Answer 1

您的代码确定。只需在末尾添加collect即可：

result.collect()  

[('s', 1),
 ('l', 2),
 ('a', 10),
 ('n', 1),
 ('t', 8),
 ('c', 1),
 ('p', 1),
 ('b', 2),
 ('w', 1),
 ('o', 2)]

您可以替换

import re
words = inRDD.flatMap(lambda l: re.split(" ",l))

与

words = inRDD.flatMap(str.split)

Answer 2

使用sparkSQL函数最简单的方法针对Apache PySpark的Wordcount程序

import pyspark.sql.functions as f

wordsDF = spark.read.text("path/log.txt")
df = wordsDF.withColumn('wordCount', f.size(f.split(f.col('value'), ' ')))
df.createOrReplaceTempView("wc")
df2 = spark.sql("SELECT SUM(wordCount) as Total FROM wc").show()


+-----+
|Total|
+-----+
|  147|
+-----+

Apache PySpark的Wordcount程序

2 个答案: