Flume:拖尾和油脂未按预期工作

时间:2019-03-31 20:40:39

标签: grep flume tail

水槽黑客!

这是我要完成的工作: 两个文件A,B都可以包含带“ hello”或不带“ hello”的行。 来自两个文件中带有“ hello”的行都应以本地文件X结尾,而没有“ hello”的行应位于文件Y中。

即使这还远未达到完美的水平(确实相当粗糙),我还是决定配置4个输入,而不是多路复用:

1)尾巴-F A | grep你好

2)尾巴-F A | grep -v你好

3)尾巴-F B | grep你好

4)尾巴-F B | grep -v你好

并将grep链接到通道c_s,将grep -v链接到通道c_o

a1.sources  = r1_s r1_o r2_s r2_o
a1.sinks    = k_s k_o 
a1.channels = c_s c_o

a1.sources.r1_s.type     = exec
a1.sources.r1_s.command  = tail -F /home/cloudera/practicaingesta/server1.log | grep hello

a1.sources.r1_o.type     = exec
a1.sources.r1_o.command  = tail -F /home/cloudera/practicaingesta/server1.log | grep -v hello

a1.sources.r2_s.type     = exec
a1.sources.r2_s.command  = tail -F /home/cloudera/practicaingesta/server2.log | grep hello

a1.sources.r2_o.type     = exec
a1.sources.r2_o.command  = tail -F /home/cloudera/practicaingesta/server2.log | grep -v hello

a1.sinks.k_s.type              = file_roll
a1.sinks.k_s.sink.directory    = /home/cloudera/practicaingesta/flume_output_hello
a1.sinks.k_s.sink.rollInterval = 0 

a1.sinks.k_o.type              = file_roll
a1.sinks.k_o.sink.directory    = /home/cloudera/practicaingesta/flume_output_no_hello
a1.sinks.k_o.sink.rollInterval = 0 

a1.channels.c_s.type                = memory
a1.channels.c_s.capacity            = 1000
a1.channels.c_s.transactionCapacity = 100

a1.channels.c_o.type                = memory
a1.channels.c_o.capacity            = 1000
a1.channels.c_o.transactionCapacity = 100

a1.sources.r1_s.channels = c_s
a1.sources.r2_s.channels = c_s
a1.sources.r1_o.channels = c_o
a1.sources.r2_o.channels = c_o
a1.sinks.k_s.channel     = c_s
a1.sinks.k_o.channel     = c_o

预期结果应为

X:仅包含打招呼的行

Y:仅包含没有问候的行

实际结果是:X和Y都包含整个文件A,B。 我在做什么错了?

0 个答案:

没有答案