Question

我遇到了Flume和/或log4j的问题。我有使用log4j和flume appender的JavaEE应用程序=一切都很好并且有效。

当我使用以下配置运行水槽时，一切都还可以。

此处有水槽配置文件

#define components of agent
a1.sources = avroSource
a1.channels = ch1
a1.sinks = hdfsSink
#define source
a1.sources.avroSource.type = avro
a1.sources.avroSource.bind = localhost
a1.sources.avroSource.port = 41415
a1.sources.avroSource.channels = ch1
#define interceptors
a1.sources.avroSource.interceptors = i1 i2 i3 i4 i5
a1.sources.avroSource.interceptors.i1.type = timestamp
a1.sources.avroSource.interceptors.i2.type = host
a1.sources.avroSource.interceptors.i3.type = regex_filter
a1.sources.avroSource.interceptors.i4.type = regex_filter
a1.sources.avroSource.interceptors.i5.type = static
a1.sources.avroSource.interceptors.i2.useIP = false
a1.sources.avroSource.interceptors.i2.hostHeader = host
a1.sources.avroSource.interceptors.i3.regex = .*AppLogging.*
a1.sources.avroSource.interceptors.i3.excludeEvents = false
a1.sources.avroSource.interceptors.i4.regex = .*;ERORR;.*
a1.sources.avroSource.interceptors.i4.excludeEvents = true
a1.sources.avroSource.interceptors.i5.key = instance
a1.sources.avroSource.interceptors.i5.value = JBT_TEST
#define channel
a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000
a1.channels.ch1.transactionCapacity = 100
#define sink
a1.sinks.hdfsSink.type = hdfs
a1.sinks.hdfsSink.channel = ch1
a1.sinks.hdfsSink.hdfs.writeFormat = Text
a1.sinks.hdfsSink.hdfs.filePrefix =  %{host}
a1.sinks.hdfsSink.hdfs.fileType = DataStream
a1.sinks.hdfsSink.hdfs.path = hdfs://192.168.79.128:9000/hdfs/mr/input/flume/%{instance}/
a1.sinks.hdfsSink.hdfs.useLocalTimeStamp = true
a1.sinks.hdfsSink.hdfs.rollInterval = 0
a1.sinks.hdfsSink.hdfs.rollCount = 0
a1.sinks.hdfsSink.hdfs.rollSize = 0
a1.sinks.hdfsSink.hdfs.batchSize = 1

如果我将接收器从hdfs更改为file_roll，则flume-agent会创建日志文件但不包含任何内容。来自水槽的新配置文件。

#define components of agent
a1.sources = avroSource
a1.channels = ch1
a1.sinks = file

#define source
a1.sources.avroSource.type = avro
a1.sources.avroSource.bind = localhost
a1.sources.avroSource.port = 41415
a1.sources.avroSource.channels = ch1

#define channel
a1.channels.ch1.type = memory
a1.channels.ch1.capacity = 1000
a1.channels.ch1.transactionCapacity = 100

#define sink
a1.sinks.file.type = file_roll
a1.sinks.file.channel = ch1
a1.sinks.file.sink.directory = c:/temp/
a1.sinks.file.sink.rollInterval = 600

我的错误是什么？

Answer 1

在配置的末尾添加以下两行，以使用通道连接源和接收器。如果有效，请告诉我。

a1.sources.avroSource.channels = ch1
a1.sinks.file.channel = ch1

Flume配置file_roll

1 个答案: