“如何在水槽中使用TAILDIR源时从存储在position_file.json中的最新关闭位置加载行”

时间:2019-01-10 09:19:13

标签: java flume

我正在使用水槽通过TAILDIR源从日志文件加载流数据。

我已经用过 myagent.sources.source_name.positionFile = /path/to/position/file

文件也正在创建中,并且每次具有最新读取位置时,但是在读取时,源文件都不从该位置读取。

打印的日志具有关闭位置和打开位置的语句以及位置,但是当添加新数据时,它将从先前关闭的位置读取,而不是从最近关闭的位置读取。

请在输出屏幕上找到配置和日志。

配置:

myagent.sources = a1
myagent.sinks = kafka-sink
myagent.channels = memoryChannel
myagent.sources.a1.channels = memoryChannel
myagent.sinks.kafka-sink.channel = memoryChannel
myagent.sources.a1.type = TAILDIR
myagent.sources.a1.filegroups = f1 
myagent.sources.a1.filegroups.f1 = /home/Documents/ecomMR/input.log
myagent.sources.a1.headers.f1.headerKey1 = value1
myagent.sources.a1.positionFile = /home/Documents/tail_position.json
myagent.sources.a1.maxBackoffSleep = 0
myagent.sources.a1.idleTimeout= 0
myagent.sources.a1.batchSize =10
myagent.sources.a1.writePosInterval= 1
myagent.sources.a1.maxBackoffSleep= 0
myagent.sources.a1.skipToEnd= true
myagent.sources.a1.backoffSleepIncrement= 0
myagent.sources.a1.fileHeader= true
myagent.sources.a1.byteOffsetHeader= true
myagent.channels.memoryChannel.type =memory
myagent.channels.memoryChannel.capacity = 200000
myagent.channels.memoryChannel.transactionCapacity = 1000
myagent.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
myagent.sinks.kafka-sink.brokerList = localhost:2181
myagent.sinks.kafka-sink.kafka.topic = kafka-flume
myagent.sinks.kafka-sink.kafka.bootstrap.servers = localhost:9092
myagent.sinks.kafka-sink.kafka.flumeBatchSize = 100
myagent.sinks.kafka-sink.kafka.producer.acks = 1
myagent.sinks.kafka-sink.kafka.producer.linger.ms = 1
myagent.sinks.kafka-sink.kafka.producer.compression.type = snappy

日志:

2019-01-10 12:40:45,618 (PollableSourceRunner-TaildirSource-a1) [INFO - org.apache.flume.source.taildir.TaildirSource.closeTailFiles(TaildirSource.java:288)] Closed file: /home/Documents/ecomMR/input.log, inode: 8524528, pos: 18

2019-01-10 12:40:58,633 (PollableSourceRunner-TaildirSource-a1) [INFO - org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:283)] Opening file: /home/Documents/ecomMR/input.log, inode: 8524533, pos: 0

2019-01-10 12:41:00,638 (PollableSourceRunner-TaildirSource-a1) [INFO - org.apache.flume.source.taildir.TaildirSource.closeTailFiles(TaildirSource.java:288)] Closed file: /home/Documents/ecomMR/input.log, inode: 8524533, pos: 24

2019-01-10 12:41:03,640 (PollableSourceRunner-TaildirSource-a1) [INFO - org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:283)] Opening file: /home/Documents/ecomMR/input.log, inode: 8524528, pos: 18

2019-01-10 12:41:05,642 (PollableSourceRunner-TaildirSource-a1) [INFO - org.apache.flume.source.taildir.TaildirSource.closeTailFiles(TaildirSource.java:288)] Closed file: /home/Documents/ecomMR/input.log, inode: 8524528, pos: 29

2019-01-10 12:42:42,692 (PollableSourceRunner-TaildirSource-a1) [INFO - org.apache.flume.source.taildir.ReliableTaildirEventReader.openFile(ReliableTaildirEventReader.java:283)] Opening file: /home/Documents/ecomMR/input.log, inode: 8524533, pos: 24

2019-01-10 12:42:45,694 (PollableSourceRunner-TaildirSource-a1) [INFO - org.apache.flume.source.taildir.TaildirSource.closeTailFiles(TaildirSource.java:288)] Closed file: /home/Documents/ecomMR/input.log, inode: 8524533, pos: 33

提前了解解决方案。

0 个答案:

没有答案