我正在尝试一个简单的练习来学习Apache Flume。我正在尝试将纯文本文件从本地文件夹移动到另一个本地文件夹,但是我希望将纯文本文件转换为avro。我的目标不是将纯文本转换为avro。我想学习Flume的工作原理。 Flume代理配置是这样的:
plain_to_avro_translator.sources = plain-source avro-source
plain_to_avro_translator.sinks = avro-sink local-file-sink
plain_to_avro_translator.channels = mem-channel1 mem-channel2
plain_to_avro_translator.sources.plain-source.channels = mem-channel1
plain_to_avro_translator.sources.plain-source.type = exec
plain_to_avro_translator.sources.plain-source.restart = true
plain_to_avro_translator.sources.plain-source.restartThrottle = 40000
plain_to_avro_translator.sources.plain-source.command = cat /home/giacobbino/data.log
plain_to_avro_translator.sinks.avro-sink.channel = mem-channel1
plain_to_avro_translator.sinks.avro-sink.type = avro
plain_to_avro_translator.sinks.avro-sink.hostname = 192.168.200.43
plain_to_avro_translator.sinks.avro-sink.port = 6000
plain_to_avro_translator.sinks.avro-sink.batch-size = 10000
plain_to_avro_translator.channels.mem-channel1.type = memory
plain_to_avro_translator.channels.mem-channel1.capacity = 100
plain_to_avro_translator.channels.mem-channel1.transactionCapacity = 100
plain_to_avro_translator.sources.avro-source.channels = mem-channel2
plain_to_avro_translator.sources.avro-source.type = avro
plain_to_avro_translator.sources.avro-source.bind = 0.0.0.0
plain_to_avro_translator.sources.avro-source.port = 6000
plain_to_avro_translator.channels.mem-channel2.type = memory
plain_to_avro_translator.channels.mem-channel2.capacity = 100
plain_to_avro_translator.channels.mem-channel2.transactionCapacity = 100
plain_to_avro_translator.sinks.local-file-sink.channel = mem-channel2
plain_to_avro_translator.sinks.local-file-sink.type = file_roll
plain_to_avro_translator.sinks.local-file-sink.sink.directory = /home/giacobbino/flume_output
plain_to_avro_translator.sinks.local-file-sink.serializer = avro_event
plain_to_avro_translator.sinks.local-file-sink.serializer.compressionCodec = snappy
我尝试编写一个遵循模式的输入json文件(该文件是正确的文件,因为我可以使用avro-tools将其转换为avro),但是得到的结果是相同的:文件按原样移动。似乎没有读取avro选项。
Flume是否可以将纯文本转换为avro,还是应该在将事件发送到源之前手动执行?
我应该添加一个具有exec源的流,然后在移动输入文件之前对输入文件执行avro-tools命令吗?
感谢您的帮助。 谢谢。