Question

我们发现了一些与感染有关的域名。现在我们在.json文件中有一个DNS名称列表，我想生成一个汇总输出，显示：用户列表，他们访问的唯一域，总计数。如果我也可以计算每个域名的数量，那么奖励积分。

以下是该文件的示例：

{"machine": "possible_victim01", "domain": "evil.com", "timestamp":1435071870}
{"machine": "possible_victim01", "domain": "evil.com", "timestamp":1435071875}
{"machine": "possible_victim01", "domain": "soevil.com", "timestamp":1435071877}
{"machine": "possible_victim02", "domain": "bad.com", "timestamp":1435071877}
{"machine": "possible_victim03", "domain": "soevil.com", "timestamp":1435071879}

理想情况下，我希望输出类似于：

{"possible_victim01": "total": 3, {"evil.com": 2, "soevil.com": 1}}
{"possible_victim02": "total": 1, {"bad.com": 1}}
{"possible_victim03": "total": 1, {"soevil.com": 1}}

我很乐意接受：

{"possible_victim01": "total": 3, ["evil.com", "soevil.com"]}
{"possible_victim02": "total": 1, ["bad.com"]}
{"possible_victim03": "total": 1, ["soevil.com"]}

我可以获得每个用户的总记录数，但是我丢失了域列表：

cat sample.json | jq -s 'group_by(.machine) | map({machine:.[0].machine,domain:.[0].domain, count:length}) '
[{"machine": "possible_victim01", "domain": "evil.com", "count": 3},  
{"machine": "possible_victim02", "domain": "bad.com", "count": 1},
{"machine": "possible_victim03", "domain": "soevil.com", "count": 1}]

这篇文章描述了如何解决问题的后半部分...... JQ Aggregations and Crosstabs。我还没有找到任何描述上半部分的内容，请到：

{"machine": "possible_victim01", "domain": "evil.com", "count":2}
{"machine": "possible_victim01", "domain": "soevil.com", "count":1}
{"machine": "possible_victim02", "domain": "bad.com", "count":1}
{"machine": "possible_victim03", "domain": "soevil.com", "count":1}

Answer 1

您需要执行group_by两次，一次按机器名称分组，然后进行子分组以获取每个域的子计数。

jq查询：

group_by(.machine) | map({
    "machine": .[0].machine, 
    "total":length, 
    "domains": (group_by(.domain) | map({
        "key":.[0].domain, 
        "value":length}) | from_entries
    )
})

示例输出：

{
  "machine": "possible_victim01",
  "total": 3,
  "domains": {
    "evil.com": 2,
    "soevil.com": 1
  }
}
{
  "machine": "possible_victim02",
  "total": 1,
  "domains": {
    "bad.com": 1
  }
}
{
  "machine": "possible_victim03",
  "total": 1,
  "domains": {
    "soevil.com": 1
  }
}

Answer 2

以所述方式使用group_by很好，但是如果你有如建议的那样读取非常多的行（即JSON实体）通过提供的示例，您可能会遇到性能问题和/或能力限制。

这些问题可以在内置“输入”的任何版本的jq中非常有效地解决（例如jq 1.5rc1）。

请注意，使用“输入”，您将使用-n选项调用jq，如下所示：

jq -n -f program.jq data.json

另请注意，此处最好生成JSON输出，以下内容似乎接近所需：

{"possible_victim01": { "total": 3, "evildoers": {"evil.com": 2, "soevil.com": 1} },
 "possible_victim02": ...}`

以下程序可以更简洁，但是此处的演示旨在使流程透明，假设对jq有基本的了解。如果这里有魔法，这是一个人不必做出“null”的特殊情况。

reduce inputs as $line
  ({};
   . as $in
   | ($line.machine) as $machine
   | ($line.domain) as $domain
   | ($in[$machine].evildoers ) as $evildoers
   | . + { ($machine): {"total": (1 + $in[$machine]["total"]),
                        "evildoers": ($evildoers | (.[$domain] += 1)) }} )

使用提供的样本输入，输出为：

{
  "possible_victim01": {
    "total": 3,
    "evildoers": {
      "evil.com": 2,
      "soevil.com": 1
    }
  },
  "possible_victim02": {
    "total": 1,
    "evildoers": {
      "bad.com": 1
    }
  },
  "possible_victim03": {
    "total": 1,
    "evildoers": {
      "soevil.com": 1
    }
  }
}

Answer 3

以下是使用reduce，getpath和setpath

的解决方案

reduce .[] as $o (
  {}
; [$o.machine, "total"] as $p1
| [$o.machine, "domains", $o.domain] as $p2
| setpath($p1; 1+getpath($p1))
| setpath($p2; 1+getpath($p2))
)

如果filter.jq包含此过滤器且data.json包含示例数据，则命令

$ jq -M -s -f filter.jq data.json

产生

{
  "possible_victim01": {
    "total": 3,
    "domains": {
      "evil.com": 2,
      "soevil.com": 1
    }
  },
  "possible_victim02": {
    "total": 1,
    "domains": {
      "bad.com": 1
    }
  },
  "possible_victim03": {
    "total": 1,
    "domains": {
      "soevil.com": 1
    }
  }
}

使用jq指望多个级别

3 个答案: