我正在尝试将PySpark中的二元组计数程序拼凑起来,该程序接收一个文本文件并输出每个正确的二元组的频率(一个句子中连续两个单词)。
from pyspark.ml.feature import NGram
with use_spark_session("Bigrams") as spark:
text_file = spark.sparkContext.textFile(text_path)
sentences = text_file.flatMap(lambda line: line.split(".")) \
.filter(lambda line: len(line) > 0) \
.map(lambda line: (0, line.strip().split(" ")))
sentences_df = sentences.toDF(schema=["id", "words"])
ngram_df = NGram(n=2, inputCol="words", outputCol="bigrams").transform(sentences_df)
ngram_df.select("bigrams")
现在包含:
+--------------------+
| bigrams|
+--------------------+
|[April is, is the...|
|[It is, is one, o...|
|[April always, al...|
|[April always, al...|
|[April's flowers,...|
|[Its birthstone, ...|
|[The meaning, mea...|
|[April comes, com...|
|[It also, also co...|
|[April begins, be...|
|[April ends, ends...|
|[In common, commo...|
|[In common, commo...|
|[In common, commo...|
|[In years, years ...|
|[In years, years ...|
+--------------------+
所以每个句子都有bigrams列表。现在需要计算不同的双字母。怎么样?此外,整个代码似乎仍然不必要地冗长,所以我很乐意看到更简洁的解决方案。
答案 0 :(得分:1)
如果您已使用RDD
API,则可以按照
bigrams = text_file.flatMap(lambda line: line.split(".")) \
.map(lambda line: line.strip().split(" ")) \
.flatMap(lambda xs: (tuple(x) for x in zip(xs, xs[1:])))
bigrams.map(lambda x: (x, 1)).reduceByKey(lambda x, y: x + y)
否则:
from pyspark.sql.functions import explode
ngram_df.select(explode("bigrams").alias("bigram")).groupBy("bigram").count()