我遵循火花流+水槽整合的指导。但我最终无法获得任何活动。 (https://spark.apache.org/docs/latest/streaming-flume-integration.html) 任何人都可以帮我分析一下吗? 在烟雾中,我创建了" avro_flume.conf"的文件。如下:
a1.sources = r1
a1.channels = c1
a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = 123.57.54.113
a1.sources.r1.port = 4141
a1.sinks = k1
a1.sinks.k1.type = avro
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k1.hostname = 123.57.54.113
a1.sinks.k1.port = 6666
a1.sources = r1
a1.sinks = spark
a1.channels = c1
在文件中,123.57.54.113是localhost的ip。
flume-ng agent -c . -f conf/avro_spark.conf -n a1 Start Spark-streaming
bin/run-example org.apache.spark.examples.streaming.FlumeEventCount 123.57.54.113 6666
flume-ng avro-client -c . -H 123.57.54.113 -p 4141 -F test/log.01
最后,根本没有任何事件。
问题是什么? 谢谢!
答案 0 :(得分:1)
我在“绑定源并接收到通道”标题下看到“a1.sinks = spark”。但名称为“spark”的接收器未在您的配置中的其他位置定义。 您是尝试从“https://spark.apache.org/docs/latest/streaming-flume-integration.html”接近1还是接近2?
如果您尝试接近1,请尝试删除“a1.sinks = spark”行。
对于方法2,请使用以下模板:
agent.sinks = spark
agent.sinks.spark.type = org.apache.spark.streaming.flume.sink.SparkSink
agent.sinks.spark.hostname = <hostname of the local machine>
agent.sinks.spark.port = <port to listen on for connection from Spark>
agent.sinks.spark.channel = memoryChannel