Kinesis Agent不会将记录转换为JSON

时间：2016-09-09 11:37:08

标签： amazon-web-services amazon-kinesis amazon-kinesis-firehose amazon-kinesis-agent

我正在使用Amazon Kinesis Agent来查找我的nginx Web访问日志，以便将访问日志推送到Kinesis - 最终将在Kinesis Analytics中使用。

代理正在运行并且正在跟踪我的访问日志，然后将记录推送到Kinesis。我可以看到到达Kinesis的记录，所以连接性很好。

我与格式化程序选项相关的问题 - 特别是关于＆＃34; LOGTOJSON ＆＃34;。

以下是/etc/aws-kinesis/agent.json的内容：

{
  "cloudwatch.emitMetrics": false,
  "kinesis.endpoint": "kinesis.eu-west-1.amazonaws.com",

  "flows": [
    {
      "filePattern": "/var/log/nginx/proxy-access*",
      "kinesisStream": "removed-xxx",
      "partitionKeyOption": "RANDOM",
      "dataProcessingOptions": {
        "optionName": "LOGTOJSON",
        "logFormat": "COMBINEDAPACHELOG"
    }
  ]
}

以下是我用于以apache组合日志格式输出日志的nginx配置部分：

log_format combined_apache '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';
access_log logs/proxy-access.log combined_apache;

以下是nginx访问日志的几行输出：

172.17.0.1 - - [09/Sep/2016:11:04:27 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
172.17.0.1 - - [09/Sep/2016:11:04:27 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
172.17.0.1 - - [09/Sep/2016:11:04:27 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
172.17.0.1 - - [09/Sep/2016:11:04:27 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"

这是Kinesis代理的输出。您可以看到记录正在被解析＆＃34;但没有＆＃34;转变＆＃34;。 

2016-09-09 11:03:20.574+0000 18b332249e3f (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 60041ms
2016-09-09 11:03:20.574+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2016-09-09 11:03:50.572+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 0 records (0 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2016-09-09 11:03:50.572+0000 18b332249e3f (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 0 records parsed (0 bytes), and 0 records sent successfully to destinations. Uptime: 90038ms
2016-09-09 11:04:20.572+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 398 records (36218 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
2016-09-09 11:04:20.572+0000 18b332249e3f (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 398 records parsed (36218 bytes), and 0 records sent successfully to destinations. Uptime: 120038ms
2016-09-09 11:04:50.572+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 533 records (48503 bytes), transformed 0 records, skipped 0 records, and has successfully sent 500 records to destination.
2016-09-09 11:04:50.572+0000 18b332249e3f (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 533 records parsed (48503 bytes), and 500 records sent successfully to destinations. Uptime: 150038ms
2016-09-09 11:05:20.571+0000 18b332249e3f (FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[kinesis:centurion-weblogs:/var/log/nginx/proxy-access*]: Tailer Progress: Tailer has parsed 533 records (48503 bytes), transformed 0 records, skipped 0 records, and has successfully sent 500 records to destination.

Kinesis Analytics将传入数据检测为CSV流而非JSON流。当我在AWS控制台中查看原始流内容时，它显示如下：

172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:58 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:59 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"
,172.17.0.1 - - [09/Sep/2016:11:03:59 +0000] "GET / HTTP/1.0" 302 337 "-" "ApacheBench/2.3"

因此，看起来代理正在发送每条记录而没有任何转换，并用逗号分隔每一行。

我无法在这里看到我出错的地方。 nginx访问日志看起来像是以正确的格式编写，但Kinesis代理并没有将其转换为JSON - 原因不明。

任何指针都会受到赞赏。

1 个答案:

答案 0 :(得分：2)

我不知道你是否已经解决了这个问题，但 dataProcessingOptions 期待一个数组（http://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html）

示例：

"dataProcessingOptions": [
        {
           other configs, etc
        }
]
相关问题
最新问题