当使用多端口syslog TCP作为源从heka推送时,Apache flume不会逐行读取日志

时间:2016-08-26 09:10:07

标签: flume hekad

我正在使用heka将原木推到水槽。 在heka我能看到日志被逐个事件标记(逐行) 但是在水槽中,我没有逐行获取。它将2行或3行日志合并在一起,并将其作为一大块消息打印出来。 我使用source作为multiport_syslogtcp和channel作为内存通道,sink作为file_roll。

Flume conf文件:

a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.sources.r1.type = multiport_syslogtcp
a1.sources.r1.host = 0.0.0.0
a1.sources.r1.ports = 44444

a1.sinks.k1.type = file_roll
a1.sinks.k1.channel = c1
a1.sinks.k1.sink.directory = /var/log/flumelog
a1.sinks.k1.sink.rollInterval = 0

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

示例日志行:

{ "name": "SIDXDropped", "host": "astr", "date": "2016-08-10T10:46:13.419Z", "schema": "mce-data", "source": "MOS_MCE_VM:worker", "type": "Application", "subtype": "", "context": "", "category": "LIVE.capture", "severity": "Critical", "description": "", "location": "node:2561", "data": {"event":{"eventData":{"channelName":"ahdenclinearwf/53560_abr","eventDetail":"RepID: 2000000_audio_frag_302 SIDX: {reference_ID: 302; PTS: 323565332463; first_offset: 72970420204; size: 821936; duration: 194400; timescale: 90000","eventName":"SIDXDropped","level":"MAJOR","profileName":"1864k","serviceType":"LIVE","subType":"CIFConverter","type":"RECORDER"}}}, "alarmop": "" }
{ "name": "SIDXDropped", "host": "astr", "date": "2016-08-10T10:46:13.419Z", "schema": "mce-data", "source": "MOS_MCE_VM:worker", "type": "Application", "subtype": "", "context": "", "category": "LIVE.capture", "severity": "Critical", "description": "", "location": "node:2561", "data": {"event":{"eventData":{"channelName":"ahdenclinearwf/53560_abr","eventDetail":"RepID: 2000000_audio_frag_302 SIDX: {reference_ID: 302; PTS: 323565332463; first_offset: 72970420204; size: 821936; duration: 194400; timescale: 90000","eventName":"SIDXDropped","level":"MAJOR","profileName":"1864k","serviceType":"LIVE","subType":"CIFConverter","type":"RECORDER"}}}, "alarmop": "" }

heka的输出示例:

2016/08/26 09:00:58 {"message":"{ \"name\": \"SIDXDropped\", \"host\": \"astr\", \"date\": \"2016-08-10T10:46:13.419Z\", \"schema\": \"mce-data\", \"source\": \"MOS_MCE_VM:worker\", \"type\": \"Application\", \"subtype\": \"\", \"context\": \"\", \"category\": \"LIVE.capture\", \"severity\": \"Critical\", \"description\": \"\", \"location\": \"node:2561\", \"data\": {\"event\":{\"eventData\":{\"channelName\":\"ahdenclinearwf\/53560_abr\",\"eventDetail\":\"RepID: 2000000_audio_frag_302 SIDX: {reference_ID: 302; PTS: 323565332463; first_offset: 72970420204; size: 821936; duration: 194400; timescale: 90000\",\"eventName\":\"SIDXDropped\",\"level\":\"MAJOR\",\"profileName\":\"1864k\",\"serviceType\":\"LIVE\",\"subType\":\"CIFConverter\",\"type\":\"RECORDER\"}}}, \"alarmop\": \"\" }","@timestamp":"2016-08-26T09:00:58.494Z","subsystem":"cls","host":"cls-1","severity":7,"component":"mce","logger":"mos-mce","logsource":"event"}
2016/08/26 09:00:58 {"message":"{ \"name\": \"SIDXDropped\", \"host\": \"astr\", \"date\": \"2016-08-10T10:46:13.419Z\", \"schema\": \"mce-data\", \"source\": \"MOS_MCE_VM:worker\", \"type\": \"Application\", \"subtype\": \"\", \"context\": \"\", \"category\": \"LIVE.capture\", \"severity\": \"Critical\", \"description\": \"\", \"location\": \"node:2561\", \"data\": {\"event\":{\"eventData\":{\"channelName\":\"ahdenclinearwf\/53560_abr\",\"eventDetail\":\"RepID: 2000000_audio_frag_302 SIDX: {reference_ID: 302; PTS: 323565332463; first_offset: 72970420204; size: 821936; duration: 194400; timescale: 90000\",\"eventName\":\"SIDXDropped\",\"level\":\"MAJOR\",\"profileName\":\"1864k\",\"serviceType\":\"LIVE\",\"subType\":\"CIFConverter\",\"type\":\"RECORDER\"}}}, \"alarmop\": \"\" }","@timestamp":"2016-08-26T09:00:58.494Z","subsystem":"cls","host":"cls-1","severity":7,"component":"mce","logger":"mos-mce","logsource":"event"}

文件中的等效消息:

{"message":"{ \"name\": \"SIDXDropped\", \"host\": \"astr\", \"date\": \"2016-08-10T10:46:13.419Z\", \"schema\": \"mce-data\", \"source\": \"MOS_MCE_VM\", \"type\": \"Application\", \"subtype\": \"\", \"context\": \"\", \"category\": \"LIVE.capture\", \"severity\": \"Critical\", \"description\": \"\", \"location\": \"node:2561\", \"data\": {\"event\":{\"eventData\":{\"channelName\":\"ahdenclinearwf\/53560_abr\",\"eventDetail\":\"RepID: 2000000_audio_frag_302 SIDX: {reference_ID: 302; PTS: 323565332463; first_offset: 72970420204; size: 821936; duration: 194400; timescale: 90000\",\"eventName\":\"SIDXDropped\",\"level\":\"MAJOR\",\"profileName\":\"1864k\",\"serviceType\":\"LIVE\",\"subType\":\"CIFConverter\",\"type\":\"RECORDER\"}}}, \"alarmop\": \"\" }","@timestamp":"2016-08-26T06:58:06.743Z","subsystem":"cls","host":"cls-1","severity":7,"component":"mce","logger":"mos-mce","logsource":"event"}{"message":"{ \"name\": \"SIDXDropped\", \"host\": \"astr\", \"date\": \"2016-08-10T10:46:13.419Z\", \"schema\": \"mce-data\", \"source\": \"MOS_MCE_VM\", \"type\": \"Application\", \"subtype\": \"\", \"context\": \"\", \"category\": \"LIVE.capture\", \"severity\": \"Critical\", \"description\": \"\", \"location\": \"node:2561\", \"data\": {\"event\":{\"eventData\":{\"channelName\":\"ahdenclinearwf\/53560_abr\",\"eventDetail\":\"RepID: 2000000_audio_frag_302 SIDX: {reference_ID: 302; PTS: 323565332463; first_offset: 72970420204; size: 821936; duration: 194400; timescale: 90000\",\"eventName\":\"SIDXDropped\",\"level\":\"MAJOR\",\"profileName\":\"1864k\",\"serviceType\":\"LIVE\",\"subType\":\"CIFConverter\",\"type\":\"RECORDER\"}}}, \"alarmop\": \"\" }","@timestamp":"2016-08-26T09:00:58.494Z","subsystem":"cls","host":"cls-1","severity":7,"component":"mce","logger":"mos-mce","logsource":"event"}

水槽中的输出日志不会按照Heka收到的行逐行打印日志。 如果有什么必须改变到水槽源或内存通道中,你能否建议

0 个答案:

没有答案