Question

我正在使用Packetbeat来监视使用端口9200上的http协议观察器进出Elasticsearch客户端节点的请求/响应。我通过Logstash发送Packetbeat的输出，然后从那里发送到Elasticsearch的另一个实例。我们在正在监控的Elasticsearch中启用了压缩支持，所以我偶尔会看到＆＃34; Accept-Encoding：gzip，deflate＆＃34;标头返回gzip压缩的响应。不幸的是，我无法使用我拥有的任何工具解码任何这些gzip响应（包括基于Web的转换器，gzip命令行工具，以及在Logstash ruby过滤器脚本中使用Zlib :: GzipReader）。他们都报告说它不是一种gzip格式。

有谁知道为什么我似乎无法解码gzip内容？

我提供了一个过滤器的示例，我在Logstash中使用它来尝试在事件通过Logstash时动态执行此操作（并且它总是报告http.response.body不是gzip格式）

filter {
  if [type] == "http" {
    if [http][response][headers][content-encoding] == "gzip" {
      ruby {
        init => "
          require 'zlib'
          require 'stringio'
        "
        code => "
          body = event.get('[http][response][body]').to_s
          sio = StringIO.new(body)
          gz = Zlib::GzipReader.new(sio)
          result = gz.read.to_s
          event.set('[http][response][body]', result)
        "
      }
    }
  }
}

我还提供了一个已记录事件的示例，其中包含gzip内容，以防您想尝试自行解压缩：

{
  "_index": "packetbeat-6.2.3-2018.05.19",
  "_type": "doc",
  "_id": "oH0bemMB2mAXfg5euIiP",
  "_score": 1,
  "_source": {
    "server": "",
    "client_server": "",
    "bytes_in": 160,
    "bytes_out": 361,
    "@timestamp": "2018-05-19T20:33:46.470Z",
    "client_port": 55863,
    "path": "/",
    "type": "http",
    "client_proc": "",
    "query": "GET /",
    "port": 9200,
    "host": "gke-main-production-elastic-clients-5728bab3-t1z8",
    "@version": "1",
    "responsetime": 0,
    "fields": {
      "nodePool": "production-elastic-clients"
    },
    "response": "HTTP/1.1 200 OK\r\ncontent-type: application/json; charset=UTF-8\r\ncontent-encoding: gzip\r\ncontent-length: 250\r\n\r\n\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
    "proc": "",
    "request": "GET / HTTP/1.1\r\nUser-Agent: vscode-restclient\r\nhost: es-http-dev.elastic-prod.svc.cluster.local:9200\r\naccept-encoding: gzip, deflate\r\nConnection: keep-alive\r\n\r\n",
    "beat": {
      "name": "gke-main-production-elastic-clients-5728bab3-t1z8",
      "version": "6.2.3",
      "hostname": "gke-main-production-elastic-clients-5728bab3-t1z8"
    },
    "status": "OK",
    "method": "GET",
    "client_ip": "10.24.20.6",
    "http": {
      "response": {
        "phrase": "OK",
        "headers": {
          "content-encoding": "gzip",
          "content-length": 250,
          "content-type": "application/json; charset=UTF-8"
        },
        "body": "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
        "code": 200
      },
      "request": {
        "params": "",
        "headers": {
          "connection": "keep-alive",
          "user-agent": "vscode-restclient",
          "content-length": 0,
          "host": "es-http-dev.elastic-prod.svc.cluster.local:9200",
          "accept-encoding": "gzip, deflate"
        }
      }
    },
    "tags": [
      "beats",
      "beats_input_raw_event"
    ],
    "ip": "10.24.41.5"
  },
  "fields": {
    "@timestamp": [
      "2018-05-19T20:33:46.470Z"
    ]
  }
}

这是我在客户端成功解压缩后在客户端收到的消息的响应：

HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 250

{
  "name": "es-client-7688c8d9b9-qp9l7",
  "cluster_name": "esprod",
  "cluster_uuid": "8iRwLMMSR72F76ZEONYcUg",
  "version": {
    "number": "5.6.3",
    "build_hash": "1a2f265",
    "build_date": "2017-10-06T20:33:39.012Z",
    "build_snapshot": false,
    "lucene_version": "6.6.1"
  },
  "tagline": "You Know, for Search"
}

Answer 1

我有不同的情况，能够解决我的问题。在此处发布，看看它是否有助于您的案例。

我使用postman工具在本地测试我的REST API服务。我的Packetbeat使用了config。

  type: http
  ports: [80, 8080, 8000, 5000, 8002]
  send_all_headers: true
  include_body_for: ["application/json", "x-www-form-urlencoded"]
  send_request: true
  send_response: true

我正在跟随身体输出。

当我在邮递员请求中添加以下内容时，我能够以明文形式获取http.response.body。

Accept-Encoding: application/json

从Packetbeat解码gzip响应体

1 个答案: