我正在尝试构建一个简单的火花流自定义接收器,其中消息直接存储在火花流中以便进行处理。但是我明白了:
java.lang.AssertionError:断言失败:Executor尚未附加到此接收器
我正在集成到第三方java库中,该库根据它正在侦听的套接字生成方法调用。通过在第三方java库中实现接口,我计划在自定义spark接收器中调用store方法。
我创建了一个简单的剪切示例,其中包含错误,但未引用第三方java库。
package com.custom.spark
import org.apache.spark.streaming.receiver.Receiver
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.storage.StorageLevel
import org.apache.spark.SparkConf
import org.apache.spark.streaming.Seconds
object CustomSparkReceiver {
def main(args: Array[String]) {
// create stream with custom receiver
val conf = new SparkConf().setMaster("local[*]").setAppName("CustomSparkReceiver")
val ssc = new StreamingContext(conf, Seconds(1))
val customReceiver = new CustomReceiver
val stream = ssc.receiverStream(customReceiver)
// print values in spark stream
stream.print
// start pushing data into stream
ssc.start
Thread.sleep(1000)
List.range(1, 10)
.foreach { number => customReceiver.store(number) }
Thread.sleep(1000)
ssc.stop()
}
class CustomReceiver extends Receiver[Integer](StorageLevel.MEMORY_AND_DISK_2) {
def onStart() = {}
def onStop() = {}
}
}
我认为这是某种线程问题但不确定如何修复它。关于上述内容的任何指示都会很棒。