在数据框中使用推送的kafka数据

时间:2019-05-23 07:22:11

标签: apache-kafka rstudio

我正在使用来自Kafka的数据,该数据我能够成功完成,但是现在我想通过将数据保存在数据帧中对其执行一些操作。我是Kafka的新手,不知道该怎么做

config <- spark_config()
config$sparklyr.shell.packages <- "org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0"
sc <- spark_connect(master = "local", config = config)

read_options  <- list(kafka.bootstrap.servers = kafka-intelligeni-cp-kafka.intelligeni-2:9092, subscribe = "metrics")
write_options <- list(kafka.bootstrap.servers = kafka-intelligeni-cp-kafka.intelligeni-2:9092, topic = "topics2")

stream <- stream_read_kafka(sc, options = read_options) %>% stream_write_kafka(options = write_options,trigger = stream_trigger_interval(20.0))

stream_view(stream)
stream_stop(stream)

0 个答案:

没有答案