使用jq

时间:2017-10-30 19:29:36

标签: json file merge jq

修改:

你好,我在合并2个文件方面遇到了问题,基本上我有2个这种结构的json文件:

[
  {
    "uri": "some/url.feature",
    "id": "safety-tests",
    "keyword": "Feature",
    "name": "Safety Tests",
    "description": "Some description",
    "line": 2,
    "tags": [
      {
        "name": "@sometag",
        "line": 1
      }
    ],
    "elements": [
      {
        "id": "some-element-id",
        "keyword": "Scenario Outline",
        "name": ": Some scenario name",
        "description": "",
        "line": 46,
        "type": "scenario",
        "tags": [
          {
            "name": "@sometag",
            "line": 1
          },
          {
            "name": "@someothertag",
            "line": 31
          }
        ],
        "before": [
          {
            "match": {
              "location": "some/test/file.rb:201"
            },
            "result": {
              "status": "passed",
              "duration": 15000
            }
          },
          {
            "match": {
              "location": "some/other/file.rb:5"
            },
            "result": {
              "status": "passed",
              "duration": 1722192000
            }
          }
        ],
        "steps": [
          {
            "keyword": "Given ",
            "name": "Some step name",
            "line": 46,
            "output": [
              "Some output"
            ],
            "match": {
              "location": "some/other/path/to/other/file.rb:137"
            },
            "result": {
              "status": "passed",
              "duration": 989158000
            }
          },
          {
            "keyword": "When ",
            "name": "some other step",
            "line": 46,
            "output": [
              "WARNING: static wait for 1 seconds."
            ],
            "match": {
              "location": "some/other/path/to/other/file.rb:80"
            },
            "result": {
              "status": "passed",
              "duration": 2700052000
            }
          },
          {
            "keyword": "And ",
            "name": "Some other name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:38"
            },
            "result": {
              "status": "passed",
              "duration": 954225000
            }
          },
          {
            "keyword": "Then ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 38792000
            }
          },
          {
            "keyword": "And ",
            "name": "And again some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 39268000
            }
          },
          {
            "keyword": "And ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 55637000
            }
          },
          {
            "keyword": "And ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 38375000
            }
          },
          {
            "keyword": "When ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:12"
            },
            "result": {
              "status": "passed",
              "duration": 751416000
            }
          },
          {
            "keyword": "And ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 28043000
            }
          },
          {
            "keyword": "Then ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:20"
            },
            "result": {
              "status": "passed",
              "duration": 5204000
            }
          }
        ],
        "after": [
          {
            "match": {
              "location": "some/other/path/to/other/file.rb:91"
            },
            "result": {
              "status": "passed",
              "duration": 20000
            }
          },
          {
            "match": {
              "location": "some/other/path/to/other/file.rb:52"
            },
            "result": {
              "status": "passed",
              "duration": 5585000
            }
          },
          {
            "match": {
              "location": "some/other/path/to/other/file.rb:27"
            },
            "result": {
              "status": "passed",
              "duration": 168146000
            }
          },
          {
            "match": {
              "location": "some/other/path/to/other/file.rb:428"
            },
            "result": {
              "status": "passed",
              "duration": 62000
            }
          }
        ]
      },
      {
        "id": "some-element-id",
        "keyword": "Scenario Outline",
        "name": ": Some scenario name",
        "description": "",
        "line": 46,
        "type": "scenario",
        "tags": [
          {
            "name": "@sometag",
            "line": 1
          },
          {
            "name": "@someothertag",
            "line": 31
          }
        ],
        "before": [
          {
            "match": {
              "location": "some/test/file.rb:201"
            },
            "result": {
              "status": "passed",
              "duration": 15000
            }
          },
          {
            "match": {
              "location": "some/other/file.rb:5"
            },
            "result": {
              "status": "passed",
              "duration": 1722192000
            }
          }
        ],
        "steps": [
          {
            "keyword": "Given ",
            "name": "Some step name",
            "line": 46,
            "output": [
              "Some output"
            ],
            "match": {
              "location": "some/other/path/to/other/file.rb:137"
            },
            "result": {
              "status": "passed",
              "duration": 989158000
            }
          },
          {
            "keyword": "When ",
            "name": "some other step",
            "line": 46,
            "output": [
              "WARNING: static wait for 1 seconds."
            ],
            "match": {
              "location": "some/other/path/to/other/file.rb:80"
            },
            "result": {
              "status": "passed",
              "duration": 2700052000
            }
          },
          {
            "keyword": "And ",
            "name": "Some other name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:38"
            },
            "result": {
              "status": "passed",
              "duration": 954225000
            }
          },
          {
            "keyword": "Then ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 38792000
            }
          },
          {
            "keyword": "And ",
            "name": "And again some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 39268000
            }
          },
          {
            "keyword": "And ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 55637000
            }
          },
          {
            "keyword": "And ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 38375000
            }
          },
          {
            "keyword": "When ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:12"
            },
            "result": {
              "status": "passed",
              "duration": 751416000
            }
          },
          {
            "keyword": "And ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:2"
            },
            "result": {
              "status": "passed",
              "duration": 28043000
            }
          },
          {
            "keyword": "Then ",
            "name": "Some other step name",
            "line": 46,
            "match": {
              "location": "some/other/path/to/other/file.rb:20"
            },
            "result": {
              "status": "passed",
              "duration": 5204000
            }
          }
        ],
        "after": [
          {
            "match": {
              "location": "some/other/path/to/other/file.rb:91"
            },
            "result": {
              "status": "passed",
              "duration": 20000
            }
          },
          {
            "match": {
              "location": "some/other/path/to/other/file.rb:52"
            },
            "result": {
              "status": "passed",
              "duration": 5585000
            }
          },
          {
            "match": {
              "location": "some/other/path/to/other/file.rb:27"
            },
            "result": {
              "status": "passed",
              "duration": 168146000
            }
          },
          {
            "match": {
              "location": "some/other/path/to/other/file.rb:428"
            },
            "result": {
              "status": "passed",
              "duration": 62000
            }
          }
        ]
      }
    ]
  }
]

其中elements可以在任一文件中包含任意数量的对象。这些是来自黄瓜的测试结果,因此文件A通常包含比文件B更多的元素,因为文件B是文件A中失败测试的重新运行。

例如。如果在第一遍中我们运行了100次测试,则文件A elements数组将包含100个具有上述格式的对象。但是,如果从这100个测试中有50个失败,则文件B elements数组将包含50个对象。我想要做的是用文件B覆盖文件A elements数组,只添加在两者中重复的元素。像

这样的东西

如果文件A有

"elements":[{a:1, b:2, c:3, d:2, e:9, f:4}]

和文件B

"elements":[{d:5}]

我想要新文件

"elements":[{a:1, b:2, c:3, d:5, e:9, f:4}]

到目前为止我已经

jq '.[].elements' path/to/file/b > path/to/new/file
jq --argfile file path/to/new/file '.[].elements += $file' path/to/file/b

将文件A中elements数组中的elements数组中包含的任何文件放在一起,但不删除其中的重复对象。

我尝试使用unique但不知道如何使用它。有什么想法吗?

在这里做了几个回答后我得到了

jq --argfile b ~/Desktop/cucumber-rerun.json '.[0].elements[4] *= $b[0].elements[0]' ~/Desktop/cucumber.json

开始工作,因为在我的实际例子中,我知道文件A中的元素4是我想用文件B中的1和唯一元素覆盖的元素。但是这对我来说不起作用,因为这两个文件都是自动生成的并且对象的顺序是未知的。

我想要一个命令,看到两个文件都比较它们,并自动检测A和B中的重复对象,并用A中的那些覆盖A中的那些

2 个答案:

答案 0 :(得分:0)

这是一个使用Object Multiplication的解决方案。假设您的数据位于A.jsonB.json

$ jq -M --argfile b B.json '.[0].elements[0] *= $b[0].elements[0]' A.json

产生

[
  {
    "uri": "https://someurl.com",
    "id": "some-id",
    "keyword": "SomeKeyword",
    "name": "Some Name",
    "description": "Some description for that test result",
    "line": 2,
    "tags": [
      {
        "name": "@sometag",
        "line": 1
      }
    ],
    "elements": [
      {
        "a": 5,
        "b": 2
      }
    ]
  }
]

如果您的数组包含更多数据,但您需要了解应如何识别相应的元素,这种方法很容易推广。

关于修订后的问题,此处是一个过滤器,用于更新A.json的对象,其中B.json的相应对象具有相同的.id

def INDEX(stream; idx_expr):
  reduce stream as $row ({};
    .[$row|idx_expr| if type != "string" then tojson else . end] |= $row);

def merge_by_id(a;b):
  if b then INDEX(a[];.id) * INDEX(b[];.id) | map(.) else a end;

  INDEX($b[];.id) as $i
| map( .elements = merge_by_id(.elements; $i[.id].elements) )

例如,如果上述过滤条件位于filter.jq,则A.json包含修订后的样本数据,而B.json包含

[
  {
    "id": "safety-tests",
    "elements": [
      {
        "id": "some-element-id",
        "description": "updated description"
      }
    ]
  }
]

命令

$ jq -M --argfile b B.json -f filter.jq A.json

生成结果

[
  {
    "uri": "some/url.feature",
    "id": "safety-tests",                      <------ top level .id
    ...
    "elements": [
      {
        "id": "some-element-id",               <------ element .id
        "keyword": "Scenario Outline",
        "name": ": Some scenario name",
        "description": "updated description",  <------ updated value
        "line": 46,
        "type": "scenario",
        ...

请注意,上述解决方案假定.id中的A.json元素是唯一的,否则merge_by_id将不会产生所需的输出。在这种情况下,以下过滤器就足够了:

def INDEX(stream; idx_expr):
  reduce stream as $row ({};
    .[$row|idx_expr| if type != "string" then tojson else . end] |= $row);

  (INDEX($b[];.id) | map_values(INDEX(.elements[];.id))) as $i
| map( $i[.id] as $o | if $o then .elements |= map($o[.id]//.) else . end )

此过滤器仅要求.id中的B.json个对象是唯一的。如果A.jsonB.json中存在非唯一元素,则可能需要更复杂的映射,然后才需要这个元素。

以下是带注释的过滤器版本:

def INDEX(stream; idx_expr):
  reduce stream as $row ({};
    .[$row|idx_expr| if type != "string" then tojson else . end] |= $row);

  # first create a lookup table for elements from B.json
  (                                         #       [{id:x, elements:[{id:y, ...}]}]
      INDEX($b[];.id)                       # -> {x: {id:x, elements:[{id:y, ...}]}..}
    | map_values(INDEX(.elements[];.id))    # -> {x: {y: {id:y, ...}}}
  ) as $i

  # update A.json objects
| map(                                      # for each object in A.json
    $i[.id] as $o                           # do we have updated values from B.json ?
  | if $o then .elements |= map($o[.id]//.) # if so then replace corresponding elements
    else . end                              # otherwise leave object unchanged
  )

答案 1 :(得分:0)

简明扼要地解决问题需要两个关键的见解:

  • 如果A和B是两个对象,那么你可以通过写下来给它们加上优先权:A + B

  • 要更新对象&#34;在适当位置&#34;,我们可以根据需要使用| =或+ =。

在您的情况下,我们可以通过以下方式应用这些见解:

$(document).width()

假设沿着以下行调用:

.[0].elements[0] += $B[0].elements[0]

输出

输入后,输出包括:

jq --argfile B B.json -f combine.jq A.json 

根据要求。

进一步说明

要使用上述jq调用产生的输出覆盖A.json,您可能希望使用"elements": [ { "a": 5, "b": 2 } ] ,但请注意风险。

如果您因任何原因不想使用sponge,可以改用--argfile,但是您必须写--slurpfile