创建StreamingContext

Question

我打开火花壳

spark-shell --packages org.apache.spark:spark-streaming-kafka_2.10:1.6.0

然后我想创建一个流式上下文

import org.apache.spark._
import org.apache.spark.streaming._


val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount").set("spark.driver.allowMultipleContexts", "true")
val ssc = new StreamingContext(conf, Seconds(1))

我遇到了一个例外：

org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243). To ignore this error, set spark.driver.allowMultipleContexts = true. The currently running SparkContext was created at:

Answer 1

当你打开spark-shell时，已经创建了一个流上下文。它被称为sc，这意味着您不需要创建配置对象。只需使用现有的sc对象。

val ssc = new StreamingContext(sc,Seconds(1))

而不是var我们将使用val