Question

我正在尝试将Spark与Kafka和Python集成在一起。我有一个名为mawstopic的主题，可打印json。我正在打印“ 名称”

的计数值

import findspark
findspark.init(r'C:\Users\maws.k\Desktop\spark-2.2.1-bin-hadoop2.7')
import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.0.2 pyspark-shell'
import sys
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

sc = SparkContext(appName="PythonStreamingDirectKafkaWordCount")
ssc = StreamingContext(sc, 2)
kafkaStream = KafkaUtils.createStream(ssc, 'cdh57-01-node-01.moffatt.me:2181', 'spark-streaming', {'mawstopic':1})
parsed = kafkaStream.map(lambda v: json.loads(v[1]))
parsed.count().map(lambda x:'words: %s' % x).pprint()
authors_dstream = parsed.map(lambda author: author['Name'])
author_counts = authors_dstream.countByValue()
author_counts.pprint()
ssc.start()
ssc.awaitTermination()

我遇到以下错误。

Py4JJavaError：调用o2767.start时发生错误。：java.lang.IllegalStateException：此JVM中只能启动一个StreamingContext。当前正在运行的StreamingContext是在org.apache.spark.streaming.api.java.JavaStreamingContext.start（JavaStreamingContext.scala：556）上启动的

在运行应用程序后我已经完成了ssc.stop（）仍然会抛出错误

将Spark与Kafka集成时抛出Py4JJavaError

0 个答案: