超出内存事务容量后,Flume无法恢复

时间:2015-10-02 14:17:18

标签: flume

我正在创建一个Flume代理的概念验证,它会在接收器不可用时缓冲事件并停止从源中消耗事件。只有当接收器再次可用时,才应处理缓冲的事件,然后源重新开始消耗。

为此,我创建了一个简单的代理,它从SpoolDir读取并写入文件。为了模拟接收器服务已关闭,我更改了文件权限,因此Flume无法写入。然后我启动Flume一些事件被缓冲在内存通道中,并且当通道容量已满时它会停止消耗事件,如预期的那样。一旦文件变得可写,接收器就能够处理事件并且Flume恢复。但是,这仅在未超过事务容量时才有效。一旦超过交易容量,Flume永远不会恢复并继续写下以下错误:

2015-10-02 14:52:51,940 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - 
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to 
deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to process transaction
    at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218)
    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.ChannelException: Take list for MemoryTransaction, 
capacity 4 full, consider committing more frequently, increasing capacity, or 
increasing thread count
    at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doTake(MemoryChannel.java:96)
    at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
    at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
    at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:191)
    ... 3 more

只要内存中缓冲的事件数超过事务容量(4),就会发生此错误。我不明白为什么,因为fileout的batchSize是1,所以它应该逐个取出事件。

这是我正在使用的配置:

agent.sources = spool-src
agent.channels = mem-channel
agent.sinks = fileout

agent.sources.spool-src.channels = mem-channel
agent.sources.spool-src.type = spooldir
agent.sources.spool-src.spoolDir = /tmp/flume-spool
agent.sources.spool-src.batchSize = 1

agent.channels.mem-channel.type = memory
agent.channels.mem-channel.capacity = 10
agent.channels.mem-channel.transactionCapacity = 4

agent.sinks.fileout.channel = mem-channel
agent.sinks.fileout.type = file_roll
agent.sinks.fileout.sink.directory = /tmp/flume-output
agent.sinks.fileout.sink.rollInterval = 0
agent.sinks.fileout.batchSize = 1

我已经使用不同的通道容量值测试了此配置。交易容量(例如3和3),但没有发现信道容量已满并且Flume能够恢复的情况。

1 个答案:

答案 0 :(得分:0)

在水槽邮件列表上,有人告诉我可能this bug影响了我的概念验证。该错误需要批量大小为100,即使它在配置中指定的也不同。我重新测试了源和放大器的测试。 sink batchSizes设置为100,内存通道transactionCapacity设置为100,容量设置为300.使用这些值,概念证明完全按预期工作。