Question

我遇到Flume内存通道问题。我运行了一个Flume代理，它淹没了内存通道，日志开始溢出＆＃34;通道已满，现在无法写入数据。源将在250毫秒后再次尝试＆＃34;

到目前为止一切顺利。我停止代理，编辑flume.conf以增加容量并再试一次。问题是，在启动时，Flume会发出同样的信息：

`16/05/14 00:21:48 INFO node.Application: Starting new configuration:
 { sourceRunners:{s1=EventDrivenSourceRunner: { source:Spool  
  Directory source s1: { spoolDir: /home/vagrant/logs } }} 
  sinkRunners:{kafka-avro-sink2=SinkRunner: { 
  policy:org.apache.flume.sink.DefaultSinkProcessor@63203b59 
  counterGroup:{ name:null counters:{} } }, kafka-avro-
   sink1=SinkRunner: { 
   policy:org.apache.flume.sink.DefaultSinkProcessor@591882e6 
  counterGroup:{ name:null counters:{} } }} channels:
  {mem1=org.apache.flume.channel.MemoryChannel{name: mem1}} }
  16/05/14 00:21:48 INFO node.Application: Starting Channel mem1
   16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup:       
  Monitored counter group for type: CHANNEL, name: mem1: Successfully  
  registered new MBean.
   16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
  Component type: CHANNEL, name: mem1 started
  16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro- 
  sink2
  16/05/14 00:21:48 INFO node.Application: Starting Sink kafka-avro-sink1
  16/05/14 00:21:48 INFO node.Application: Starting Source s1
  16/05/14 00:21:48 INFO source.SpoolDirectorySource:  
  SpoolDirectorySource source starting with directory:  
  /home/vagrant/logs
  16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
  Monitored counter group for type: SOURCE, name: s1: Successfully  
  registered new MBean.
  16/05/14 00:21:48 INFO instrumentation.MonitoredCounterGroup: 
   Component type: SOURCE, name: s1 started
  16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is  
  full, and cannot write data now. The source will try again after 
  250 milliseconds
  16/05/14 00:21:49 INFO avro.ReliableSpoolingFileEventReader: Last  
  read was never committed - resetting mark position.
  16/05/14 00:21:49 WARN source.SpoolDirectorySource: The channel is  
  full, and cannot write data now. The source will try again after  
  500 milliseconds`

因此，当我启动代理时，频道已满。如何手动重置或清除它？

我搜索了几个小时类似的问题是徒劳的。一个稍微令人激动的问题是，https://flume.apache.org/是如此充满文本，它基本上被谷歌索引几乎任何与水槽有关的东西。所以对'＆＃34;水槽频道的搜索总是满满的＃34;例如，总是将https://flume.apache.org/作为第一结果。

Answer 1

似乎您无法为频道分配足够的内存。您的传入事件速率高于内存通道可用内存的位置。

尝试使用以下示例： xmx是示例选项，您可以逐步增加并根据运行parllely的代理程序数量查看内存分配的最佳值。

对于Ex：

flume-ng agent -n $agentnumber -c ../../config/conf/ -f ../../config/conf/youragentconf.conf -Xmx3g

此外，您还需要通过引用this Memory channel from Flume Guide来检查代理配置文件中的其他配置参数。

启动时Flume内存通道已满

1 个答案: