如何从数组中的数组中提取数据

时间:2018-05-31 15:22:15

标签: json csv parsing jq data-extraction

我有JSON:

{
  "count": 6918,
  "next": "https://otx.alienvault.com/api/v1/pulses/subscribed?limit=50&page=2",
  "results": [
    {
      "industries": [],
      "tlp": "white",
      "description": "With the massive ransomware campaigns of 2016 and 2017 taking a backseat to bankers and other malware families, information stealers made up 18% of malicious email payloads in the first part of this year. Proofpoint researchers recently discovered a new stealer, dubbed “Nocturnal Stealer,” most notable as an example of inexpensive commodity malware with significant potential for monetization.\n\nOn March 9, a user posted an advertisement for Nocturnal Stealer on an underground forum. The stealer sold for 1500 Rubles, or roughly US$25 at the time of analysis. Nocturnal Stealer is designed to steal the data found within multiple Chromium and Firefox based browsers. It can also steal many popular cryptocurrency wallets as well as any saved FTP passwords within FileZilla. Proofpoint researchers analyzed a sample being dropped in the wild by an unknown loader.",
      "created": "2018-05-31T12:25:14.636000",
      "tags": [],
      "modified": "2018-05-31T12:25:14.636000",
      "author_name": "AlienVault",
      "public": 1,
      "extract_source": [],
      "references": [
        "https://www.proofpoint.com/us/threat-insight/post/thief-night-new-nocturnal-stealer-grabs-data-cheap"
      ],
      "targeted_countries": [],
      "indicators": [
        {
          "indicator": "http://nctrnl.us/",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "URL",
          "id": 981584318
        },
        {
          "indicator": "http://nctrnl.us/ara.exe",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "URL",
          "id": 981584319
        },
        {
          "indicator": "http://nctrnl.us/ark.exe",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "URL",
          "id": 981584320
        },
        {
          "indicator": "c9a834dde38c8b559d575ac61046e3a3fada97d2953d902b74cf8d5e51ada30f",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "FileHash-SHA256",
          "id": 981584321
        },
        {
          "indicator": "nctrnl.us",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "domain",
          "id": 981584322
        },
        {
          "indicator": "205def439aeb685d5a9123613e49f59d4cd5ebab9e933a1567a2f2972bda18c3",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "FileHash-SHA256",
          "id": 981584323
        },
        {
          "indicator": "ae7e5a7b34dc216e9da384fcf9868ab2c1a1d731f583f893b2d2d4009da15a4e",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "FileHash-SHA256",
          "id": 981584324
        },
        {
          "indicator": "http://nctrnl.us/server/gate.php",
          "description": "",
          "created": "2018-05-31T12:25:16",
          "title": "",
          "content": "",
          "type": "URL",
          "id": 981584325
        }
      ],
      "more_indicators": false,
      "revision": 1,
      "adversary": "",
      "id": "5b0fe9aa8bc6a5498565929a",
      "name": "Thief in the night: New Nocturnal Stealer grabs data on the cheap"
    },
    ...
  ]
}

主阵列为results,我想从name数组中提取字段indicator,然后是字段typeindicators。< / p>

我使用了JQ命令:

$ jq -r '.results[] | [.name] | .indicators[] | [.type, .indicator] | @csv' \
    < /home/threat-intel/ThreatIntel/AV.json \
    >> /home/threat-intel/ThreatIntel/AV.csv

命令失败说:

jq: error (at <stdin>:0): Cannot index array with string "indicators"

如何提取name字段并将其分配给该特定数组中的每个indicator

3 个答案:

答案 0 :(得分:1)

jq 解决方案:

jq -r '.results[] | .name as $n | .indicators[] | [$n, .type, .indicator] | @csv' file.json

输出:

"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/ara.exe"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/ark.exe"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","c9a834dde38c8b559d575ac61046e3a3fada97d2953d902b74cf8d5e51ada30f"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","domain","nctrnl.us"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","205def439aeb685d5a9123613e49f59d4cd5ebab9e933a1567a2f2972bda18c3"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","ae7e5a7b34dc216e9da384fcf9868ab2c1a1d731f583f893b2d2d4009da15a4e"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/server/gate.php"

答案 1 :(得分:0)

[.name]是一个数组,因此当您将其传递到.indicators时,会收到错误消息:

  

无法使用字符串“indicators”索引数组

您的查询的这种变体可能就是您的意图:

.results[]
| .name as $name 
| .indicators[]
| [$name, .type, .indicator] | @csv

这包括每行的相关.name

"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/ara.exe"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/ark.exe"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","c9a834dde38c8b559d575ac61046e3a3fada97d2953d902b74cf8d5e51ada30f"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","domain","nctrnl.us"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","205def439aeb685d5a9123613e49f59d4cd5ebab9e933a1567a2f2972bda18c3"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","FileHash-SHA256","ae7e5a7b34dc216e9da384fcf9868ab2c1a1d731f583f893b2d2d4009da15a4e"
"Thief in the night: New Nocturnal Stealer grabs data on the cheap","URL","http://nctrnl.us/server/gate.php"

答案 2 :(得分:0)

有很多方法可以实现。从子数组中获取值以生成值时,我喜欢使用foreach来实现此目的。

$ jq -r '.results[] | foreach .indicators[] as $i ({name}; .;
    [.name,$i.type,$i.indicator]
) | @csv'