Question

我正在尝试使用Kafka作为源和水槽来摄取实时数据，因为sink.Sink类型是HDFS。我的生产者工作正常，我可以看到生成的数据和我的代理运行正常（运行命令时没有错误）但是文件没有在指定的目录中生成。

启动水槽代理的命令：

/usr/hdp/2.5.0.0-1245/flume/bin/flume-ng agent -c /usr/hdp/2.5.0.0-1245/flume/conf -f /usr/hdp/2.5.0.0-1245/flume/conf/flume-hdfs.conf -n tier1

我的flume-hdfs.conf文件：

 tier1.sources  = source1
 tier1.channels = channel1
 tier1.sinks = sink1

 tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
 tier1.sources.source1.zookeeperConnect = localhost:2181
 tier1.sources.source1.topic = data_1
 tier1.sources.source1.channels = channel1

 tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
 tier1.channels.channel1.brokerList = localhost:6667
 tier1.channels.channel1.zookeeperConnect = localhost:2181
 tier1.channels.channel1.capacity = 10000
 tier1.channels.channel1.transactionCapacity = 1000

 tier1.sinks.sink1.type = hdfs
 tier1.sinks.sink1.hdfs.path = /user/user_name/FLUME_LOGS/
 tier1.sinks.sink1.hdfs.rollInterval = 5
 tier1.sinks.sink1.hdfs.rollSize = 0
 tier1.sinks.sink1.hdfs.rollCount = 0
 tier1.sinks.sink1.hdfs.fileType = DataStream
 tier1.sinks.sink1.channel = channel1

我无法找出执行的错误。

请建议如何克服这个问题。

Answer 1

以这种方式设置HDFS接收器的路径：

tier1.sinks.sink1.hdfs.path = "VALUE of fs.default.name, located in core-site.xml"/user/user_name/FLUME_LOGS/

例如

tier1.sinks.sink1.hdfs.path = hdfs://localhost:54310/user/user_name/FLUME_LOGS/

FlumeData文件未在HDFS接收器中创建

1 个答案: