Question

我有大量的ES日志。大多数日志条目如下所示：

<input type="text" class="number-only">
<input type="text" class="number-only">
<input type="text" class="number-only">
<input type="text" class="number-only">
<input type="text" class="number-only">
<input type="text" class="number-only">

任何人都可以帮我理解吗？ - ES每分钟产生1GB！

Answer 1

您似乎正在使用某种依赖于旧版Jetty 7的ES插件。您所看到的是传入请求对Jetty来说太大而无法处理的结果（即HTTP 413请求实体太大）。 This bug出现在Jetty 7.6.3及更早版本中，并已在7.6.4版本中修复。

巨大的日志行实际上包含Jetty请求缓冲区的状态以及十六进制编码的缓冲区的当前内容。它可以帮助解码十六进制字符串，因为我们可以深入了解导致问题的原因。您似乎正在发送包含base64数据的文档。例如，您发送的其中一个文档就是这个文档的最后一个meta字段和{"base64":"eyJjb21tZW50IjoiU2VuaW9yIFNhbGVzIEV4ZWN1dGl2ZSwgT3JhY2xlIExpbnV4ICZhbXA7IFZpcnR1YWxpemF0aW9uIC0gT3JhY2xlOiAoI1NhbkZyYW5jaXNjbywgQ0EpIGh0dHA6Ly90LmNvL3JhQ3BMT2szc3AgI1NhbGVzICNKb2IgI0pvYnMgI0hpcmluZyIsInJlbW92ZWRCeVR3aXR0ZXJTdHJlYW0iOiJ5ZXMiLCJ0d2l0dGVyQXV0aG9ySUQiOjIwODI3MjA0LCJpbnNlcnRFcG9jaCI6MTQ0MzAzMTU1OC4zNzAxMDksInVwZGF0ZUVwb2NoIjoxNDQzMDMxNzc4LjgzNTkzNiwiaWQiOjE2NTY0LCJmcm9tVVJMIjoiaHR0cDovL3d3dy50d2l0dGVyLmNvbS90bWpfc2ZvX3NhbGVzL3N0YXR1cy82NDY3NDczMzIxMDk4NjA4NjQiLCJhdXRvSW5zZXJ0QnkiOiJ0d2l0dGVyU3RyZWFtIiwidm90ZVVwQ291bnQiOjAsInR5cGUiOiJjb21tZW50IiwidXNlcm5hbWUiOiJTRiBTYWxlcyBKb2JzIiwidHdpdHRlclByb2ZpbGVJbWFnZVVybCI6Imh0dHBzOi8vcGJzLnR3aW1nLmNvbS9wcm9maWxlX2ltYWdlcy8yMzAzNzUwMTAxL0xvZ29fdG1qX25ldzJiX25vcm1hbC5wbmciLCJ0d2VldExhbmd1YWdlQ29kZSI6ImVuIiwidHdlZXRHZW8iOnsidHlwZSI6IlBvaW50IiwiY29vcmRpbmF0ZXMiOlszNy43NzQ5MywtMTIyLjQxOTQxNl19LCJoYXNodGFnIjoiI3NhbmZyYW5jaXNjbyIsInZvdGVUb3RhbENvdW50IjowLCJ0d2VldElEIjoiNjQ2NzQ3MzMyMTA5ODYwODY0IiwidHdpdHRlclVzZXIiOiJ0bWpfc2ZvX3NhbGVzIiwicmVtb3ZlIjoieWVzIiwic3VidHlwZSI6InR3ZWV0IiwidGxkIjoidHdpdHRlci5jb20iLCJ2b3RlRG93bkNvdW50IjowLCJtZW50aW9ucyI6W10sInVzZXJJRCI6bnVsbH0=","meta":{"expiration":0,"flags":33554432,"id":"comment::#sanfrancisco::16564","rev":"2-00088ea2e7f9be400000000002000000"}},字段：

base64

解码{ "comment": "Senior Sales Executive, Oracle Linux & Virtualization - Oracle: (#SanFrancisco, CA) http://t.co/raCpLOk3sp #Sales #Job #Jobs #Hiring", "removedByTwitterStream": "yes", "twitterAuthorID": 20827204, "insertEpoch": 1443031558.370109, "updateEpoch": 1443031778.835936, "id": 16564, "fromURL": "http://www.twitter.com/tmj_sfo_sales/status/646747332109860864", "autoInsertBy": "twitterStream", "voteUpCount": 0, "type": "comment", "username": "SF Sales Jobs", "twitterProfileImageUrl": "https://pbs.twimg.com/profile_images/2303750101/Logo_tmj_new2b_normal.png", "tweetLanguageCode": "en", "tweetGeo": { "type": "Point", "coordinates": [ 37.77493, -122.419416 ] }, "hashtag": "#sanfrancisco", "voteTotalCount": 0, "tweetID": "646747332109860864", "twitterUser": "tmj_sfo_sales", "remove": "yes", "subtype": "tweet", "tld": "twitter.com", "voteDownCount": 0, "mentions": [], "userID": null }字段会生成此文档，该文档似乎是来自Twitter API的某种推文。

{{1}}

所以，你有两个选择：

您需要弄清楚ES安装中使用Jetty的内容，并确保至少升级到Jetty 7.6.4
您可能需要一次发送少量文档，如果您使用批量请求，可能会降低每个批量中包含的文档数量。

Elasticsearch日志条目说明

1 个答案: