Question

我想在JSON文件中执行2个操作。我试图用JQ和SHELL做到这一点。

第一个：我想将父元素转换为纯文本值

第二个：我想在JSON树中删除一个特定级别

输入：

    {
  "template_first": {
    "order": 0,
    "index_patterns": [
      "first"
    ],
    "settings": {
      "index": {
        "codec": "best_compression",
        "refresh_interval": "30s",
        "analysis": {
          "normalizer": {
            "norm_case_insensitive": {
              "filter": "lowercase",
              "type": "custom"
            }
          }
        },
        "number_of_shards": "1",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_doc": {
        "dynamic": true,
        "dynamic_templates": [
          {
            "strings": {
              "mapping": {
                "type": "keyword"
              },
              "match_mapping_type": "string"
            }
          }
        ],
        "properties": {
          "log.id": {
            "type": "keyword"
          },
          "host.indexer.hostname": {
            "type": "keyword"
          },
          "ts_indexer": {
            "format": "strict_date_optional_time||epoch_millis",
            "type": "date"
          }
        }
      }
    }
  },
  "template_second": {
    "order": 0,
    "index_patterns": [
      "second"
    ],
    "settings": {
      "index": {
        "codec": "best_compression",
        "refresh_interval": "30s",
        "analysis": {
          "normalizer": {
            "norm_case_insensitive": {
              "filter": "lowercase",
              "type": "custom"
            }
          }
        },
        "number_of_shards": "1",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_doc": {
        "dynamic": true,
        "dynamic_templates": [
          {
            "strings": {
              "mapping": {
                "type": "keyword"
              },
              "match_mapping_type": "string"
            }
          }
        ],
        "properties": {
          "log.id": {
            "type": "keyword"
          },
          "host.indexer.hostname": {
            "type": "keyword"
          },
          "ts_indexer": {
            "format": "strict_date_optional_time||epoch_millis",
            "type": "date"
          }
        }
      }
    }
  }
}

您在文件中看到两个JSON对象

    {
    "template_first" : { ...},
    "template_second" : { ... }
     }

第一个修改来自此命令的出现

输入_template / template_number

代替第一个JSON对象的键。

所以预期的结果

PUT _template/template_first
  {...}
PUT _template/template_second
  {...}

第二个更改是由于删除了_doc级别

之前：

  "mappings": {
    "_doc": {
      "dynamic": true,
      "dynamic_templates": [
        {
          "strings": {
            "mapping": {
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        }
      ],
      "properties": {
        "log.id": {
          "type": "keyword"
        },
        "host.indexer.hostname": {
          "type": "keyword"
        },
        "ts_indexer": {
          "format": "strict_date_optional_time||epoch_millis",
          "type": "date"
        }
      }
    }
  }

预期结果

  "mappings": {
    "dynamic": true,
    "dynamic_templates": [
      {
        "strings": {
          "mapping": {
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "log.id": {
        "type": "keyword"
      },
      "host.indexer.hostname": {
        "type": "keyword"
      },
      "ts_indexer": {
        "format": "strict_date_optional_time||epoch_millis",
        "type": "date"
      }
    }
  }

所以实际结果看起来像这样

PUT _template/template_first
  {
  "order": 0,
  "index_patterns": [
    "first"
  ],
  "settings": {
    "index": {
      "codec": "best_compression",
      "refresh_interval": "30s",
      "analysis": {
        "normalizer": {
          "norm_case_insensitive": {
            "filter": "lowercase",
            "type": "custom"
          }
        }
      },
      "number_of_shards": "1",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "dynamic": true,
    "dynamic_templates": [
      {
        "strings": {
          "mapping": {
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "log.id": {
        "type": "keyword"
      },
      "host.indexer.hostname": {
        "type": "keyword"
      },
      "ts_indexer": {
        "format": "strict_date_optional_time||epoch_millis",
        "type": "date"
      }
    }
  }
}
PUT _template/template_second
  {
  "order": 0,
  "index_patterns": [
    "second"
  ],
  "settings": {
    "index": {
      "codec": "best_compression",
      "refresh_interval": "30s",
      "analysis": {
        "normalizer": {
          "norm_case_insensitive": {
            "filter": "lowercase",
            "type": "custom"
          }
        }
      },
      "number_of_shards": "1",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "dynamic": true,
    "dynamic_templates": [
      {
        "strings": {
          "mapping": {
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],
    "properties": {
      "log.id": {
        "type": "keyword"
      },
      "host.indexer.hostname": {
        "type": "keyword"
      },
      "ts_indexer": {
        "format": "strict_date_optional_time||epoch_millis",
        "type": "date"
      }
    }
  }
}

我实现了第二个更改：使用命令删除JSON数组的一级

jq  'keys[] as $k | map( .mappings =.mappings._doc   )' template.json

但是我不知道如何同时进行第一次更改和第二次更改。

我试图像这样循环进入数组，但没有成功

for row in $(jq 'keys[] as $k | "\($k)"' template.json); do
    _jq() {
     echo ${row} 
    }
   echo $(_jq '.name')
done

Answer 1

仅调用一次jq，并让其编写一个以NUL分隔的模板名称/修改后的模板内容对列表（bash while read循环然后可以对其进行迭代）：

while IFS= read -r -d '' template_name && IFS= read -r -d '' template_content; do
  echo "We want to do PUT the following to _template/$template_name"
  printf '%s\n' "$template_content"
done < <(
  jq -j '
    to_entries[] |
    .key as $template_name |
    .value as $template_content |
    ($template_name, "\u0000",
     ($template_content | (.mappings = .mappings._doc) | tojson), "\u0000")
  ' <infile.json
)

Answer 2

我在done < <(上遇到了麻烦，导致我的shell中出现语法错误（不知道为什么）。所以我这样修改了您的脚本：

jq -j 'to_entries[] | .key as $template_name | .value as $template_content | ($template_name, "\u0000", ($template_content | (.mappings = .mappings._doc) | tojson), "\u0000")' < infile.json |
while IFS= read -r -d '' template_name && IFS= read -r -d '' template_content; do
  echo "PUT _template/$template_name"
  printf '%s\n' "$template_content"
done

哪个工作做得完美！谢谢查尔斯

从单个输入生成多个JQ输出文档，修改每个结果

2 个答案: