Question

我有一个这样的JSON blob：

{
    "total_items": 2371,
    "page_count": 3,
    "items": [
        {
            "landing_id": "503a654cfaf5614b2069de304dbdb3c9",
            "token": "503a654cfaf5614b2069de304dbdb3c9",
            "response_id": "503a654cfaf5614b2069de304dbdb3c9",
            "landed_at": "2019-03-07T18:35:06Z",
            "submitted_at": "2019-03-07T18:36:17Z",
            "metadata": {
                "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.2 Safari/605.1.15",
                "platform": "other",
                "network_id": "321c8d9867",
                "browser": "default"
            },
            "answers": [
                {
                    "field": {
                        "id": "uqZV4j2RSapJ",
                        "type": "opinion_scale",
                        "ref": "bed55d05-b5e2-46dd-af91-337519830e3e"
                    },
                    "type": "number",
                    "number": 5
                }
            ],
        },
        {
            "landing_id": "b9d50e6734d30753de42b78a2ad5d741",
            "token": "b9d50e6734d30753de42b78a2ad5d741",
            "response_id": "b9d50e6734d30753de42b78a2ad5d741",
            "landed_at": "2019-03-07T18:12:29Z",
            "submitted_at": "2019-03-07T18:13:54Z",
            "metadata": {
                "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1",
                "platform": "mobile",
                "network_id": "3a74e2b874",
                "browser": "touch"
            },
            "answers": [
                {
                    "field": {
                        "id": "uqZV4j2RSapJ",
                        "type": "opinion_scale",
                        "ref": "bed55d05-b5e2-46dd-af91-337519830e3e"
                    },
                    "type": "number",
                    "number": 1
                },
                {
                    "field": {
                        "id": "v8nQgm61VdLp",
                        "type": "dropdown",
                        "ref": "d20b5f19-dc7c-4497-b798-239924054367"
                    },
                    "type": "text",
                    "text": "Inconsistent information"
                }
            ],
        },
        {
            "landing_id": "96f601b5769bacf7208b1a9c8c390278",
            "token": "96f601b5769bacf7208b1a9c8c390278",
            "response_id": "96f601b5769bacf7208b1a9c8c390278",
            "landed_at": "2019-03-07T18:09:08Z",
            "submitted_at": "2019-03-07T18:09:40Z",
            "metadata": {
                "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1",
                "platform": "mobile",
                "network_id": "1c59eb873c",
                "browser": "touch"
            },
            "answers": [
                {
                    "field": {
                        "id": "uqZV4j2RSapJ",
                        "type": "opinion_scale",
                        "ref": "bed55d05-b5e2-46dd-af91-337519830e3e"
                    },
                    "type": "number",
                    "number": 5
                },
                {
                    "field": {
                        "id": "llTPythU272I",
                        "type": "multiple_choice",
                        "ref": "bf54c521-3749-4c80-93bf-bb5448d03b34"
                    },
                    "type": "choices",
                    "choices": {
                        "labels": [
                            "Travis"
                        ]
                    }
                }
            ],
        }
    ]
}

我整理了一个看起来像这样的脚本：

curl https://api.xyz.com/forms/Te8Fy9/responses?page_size=1000 -H 'Authorization: Bearer xxxxxx' -H 'cache-control: no-cache' > raw.json

page_parameter=`cat raw.json | jq -c '.items | last | .token' | tr -d '"'`

items_parameter=`cat raw.json | jq -c '.total_items'`

cat raw.json | jq -c '.items[]' > clean.json

bq load --source_format NEWLINE_DELIMITED_JSON --autodetect bq-test:sandox.test clean.json

while [[ ${#items_parameter} -gt 0 ]]
do

    curl 'https://api.xyz.com/forms/Te8Fy9/responses?page_size=1000&after='${page_parameter} -H 'Authorization: Bearer xxxxxx' -H 'cache-control: no-cache' > raw.json

    page_parameter=`cat returns_raw.json | jq -c '.items | last | .token' | tr -d '"'`

    items_parameter=`cat returns_raw.json | jq -c '.total_items'`

    cat returns_raw.json | jq -c '.items[]' > clean.json

    if [[  ${#items_parameter} -gt 0 ]]; then

        bq load --source_format NEWLINE_DELIMITED_JSON --autodetect bq-test:sandox.test clean.json

    fi

done

所以我要做的是：

1。）运行API调用（curl https...），存储为raw.json

2。）存储指定的值，即token的最后一个值作为参数。在此示例中，该值为96f601b5769bacf7208b1a9c8c390278，并存储为page_parameter

3。）存储另一个指定的值，该值与total_items关联的值位于顶部。在此示例中，该值为2371并存储为items_parameter

4。）使用jq -c '.items[]'清理raw.json并将其创建为clean.json

5。）推送至BigQuery

6。）假设它运行正确，它应该产生一个大于0的items_parameter，如果是这种情况，则取page_parameter的值来遍历子curl命令提取下1000条记录

7。）重复步骤＃2（page_parameter），步骤＃3（items_parameter）和步骤＃4（清除raw.json并另存为clean.json）< / p>

8。）如果items_parameter值大于0，则将其加载到BigQuery

因此，希望它会一直执行到该值等于0为止，此时它将退出循环。但这不是那样。它只是连续迭代相同的记录。

从理论上讲，它应该运行三个不同的记录周期，分别被加载：1-1000、1001-2000、2001-2371，然后将其断开。

我想念什么？我认为我已经很接近了，可能只是做了明显错误的事情。

Answer 1

您似乎有一个简单的错字：您将curl的结果输出为bash: ng: command not found，但使用raw.json作为进一步JQ命令的输入。如果您在所有位置都固定了文件名，它可以开始工作吗？

Bash循环不会停止循环

1 个答案: