Question

我正在尝试使用Flume-ng（1.2）将数据从平面文件（日志文件）加载到Hbase中。平面文件有多个列，每个列都是冒号（:)分开，它们都需要加载到HBASE中的单独列中。我正在检查论坛，我发现有一个来自Apache的jar来解决这个问题（org.apache.flume.sink.hbase.RegexHbaseEventSerializer），但是我无法在互联网上找到任何配置文件或用法。如果有人可以帮我配置文件，那将会很有帮助

平面文件中的内容 1：NN 2：PP 3：毫米

由于

Answer 1

RegexHbaseEventSerializer有三个可以设置的配置参数（as described in the source code）;这些是：

/** Regular expression used to parse groups from event data. */
public static final String REGEX_CONFIG = "regex";

/** Whether to ignore case when performing regex matches. */
public static final String IGNORE_CASE_CONFIG = "regexIgnoreCase";

/** Comma separated list of column names to place matching groups in. */
public static final String COL_NAME_CONFIG = "colNames";

使用RegexHbaseEventSerializer的示例配置如下（部分引用Cloudera's Flume and HBase presentation）：

host1.sources = src1
host1.sinks = sink1
host1.channels = ch1

host1.sources.src1.type = seq
host1.sources.src1.port = 25001
host1.sources.src1.bind = localhost
host1.sources.src1.channels = ch1

host1.sinks.sink1.type = org.apache.flume.sink.hbase.HBaseSink
host1.sinks.sink1.channel = ch1
host1.sinks.sink1.table = test3
host1.sinks.sink1.columnFamily = testing

host1.sinks.sink1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer
host1.sinks.sink1.serializer.regex = X
host1.sinks.sink1.serializer.regexIgnoreCase = true
host1.sinks.sink1.serializer.colNames = column_1,column_2,column_3

host1.channels.ch1.type=memory10

FlumeNG中的正则表达式配置

1 个答案: