使用jq

时间:2016-12-12 01:02:52

标签: amazon-s3 amazon-redshift jq array-merge

我有多个s3清单文件,每个文件对应于给定日期范围的日期。我希望合并所有清单文件以生成单个清单文件,从而允许我执行单个Redshift副本。

清单文件1:

{
    "entries": [
        {
            "url": "DFA/20161001/394007-OMD-Coles/dcm_account394007_activity_20160930_20161001_050403_294198927.csv.gz"
        }
    ]
}

清单文件2:

{
    "entries": [
        {
            "url": "DFA/20161002/394007-OMD-Coles/dcm_account394007_activity_20161001_20161002_054043_294865863.csv.gz"
        }
    ]
}

我正在寻找输出: -

{
    "entries": [
         {
            "url": "DFA/20161001/394007-OMD-Coles/dcm_account394007_activity_20160930_20161001_050403_294198927.csv.gz"
         },
         {
            "url": "DFA/20161002/394007-OMD-Coles/dcm_account394007_activity_20161001_20161002_054043_294865863.csv.gz"
         }
    ]
}

我确实尝试了

jq -s '.[]' "manifest_file1.json" "manifest_file2.json" 

以及Stackoverflow中发布的其他建议但无法使其正常工作。

2 个答案:

答案 0 :(得分:1)

或者,不依靠reduce

$ jq -n '{entries: [inputs.entries[]]}' manifest_file_{1,2}.json
{
  "entries": [
    {
      "url": "DFA/20161001/394007-OMD-Coles/dcm_account394007_activity_20160930_20161001_050403_294198927.csv.gz"
    },
    {
      "url": "DFA/20161002/394007-OMD-Coles/dcm_account394007_activity_20161001_20161002_054043_294865863.csv.gz"
    }
 ]
}

请注意,inputs是在jq 1.5版中引入的。如果您的jq没有inputs,则可以使用jq -s,如下所示:

$ jq -s '{entries: [.[].entries[]]}' manifest_file_{1,2}.json

答案 1 :(得分:0)

所以如果通过"合并"你的意思是通过连接它们将"entries"数组组合成一个数组,你可以这样做:

$ jq 'reduce inputs as $i (.; .entries += $i.entries)' manifest_file{1,2}.json

哪个收益率:

{
  "entries": [
    {
      "url": "DFA/20161001/394007-OMD-Coles/dcm_account394007_activity_20160930_20161001_050403_294198927.csv.gz"
    },
    {
      "url": "DFA/20161002/394007-OMD-Coles/dcm_account394007_activity_20161001_20161002_054043_294865863.csv.gz"
    }
  ]
}