多个URL保存JSON数据

时间:2019-03-20 14:06:57

标签: python python-3.x python-requests

我试图一次调用多个(超过10个网址)并保存所有这10个网址的数据,这些数据将采用json格式并尝试保存在我的位置

这是我尝试过的以下代码,使用该代码我只能获得仅保存在json文件中的最后一个URL的数据。如何获取所有URL的数据并存储在单个json文件中?

import json
import requests

URLs = ['http://httpbin.org/ip',
'http://httpbin.org/user-agent',
'http://httpbin.org/headers']

json_list = []
for url in URLs:
    data = requests.get(url)
    resolvedwo = data.json()
    with open('resolvedworesolution.json', 'w') as f:
         json.dump(resolvedwo, f)

5 个答案:

答案 0 :(得分:1)

您的问题是每次循环都覆盖文件。而是将循环结果存储在列表中,并将其仅写入文件一次

import requests
import json

URLs = ['http://httpbin.org/ip',
        'http://httpbin.org/user-agent',
        'http://httpbin.org/headers']

json_list = []

for url in URLs:
    data = requests.get(url)
    resolvedwo = data.json()
    json_list.append(resolvedwo)

with open('resolvedworesolution.json', 'w+') as f:
    json.dump(json_list, f, sort_keys=True, indent=4)

输出:

[
    {
        "origin": "137.221.143.66, 137.221.143.66"
    },
    {
        "user-agent": "python-requests/2.21.0"
    },
    {
        "headers": {
            "Accept": "*/*",
            "Accept-Encoding": "gzip, deflate",
            "Host": "httpbin.org",
            "User-Agent": "python-requests/2.21.0"
        }
    }
]

答案 1 :(得分:0)

在写入文件时使用append mode,以“保留”现有数据:

import json
import requests
URLs = ['http://httpbin.org/ip',
'http://httpbin.org/user-agent',
'http://httpbin.org/headers']

json_list = []
for url in URLs:
    data = requests.get(url)
    resolvedwo = data.json()
    with open('resolvedworesolution.json', 'a') as f:   # Using the append mode
        json.dump(resolvedwo, f)
        f.write("\n")                                   # new line for readability

输出

{"origin": "159.122.207.241, 159.122.207.241"}
{"user-agent": "python-requests/2.21.0"}
{"headers": {"Accept": "*/*", "Accept-Encoding": "gzip, deflate", "Host": "httpbin.org", "User-Agent": "python-requests/2.21.0"}}

编辑

您可以一次将响应写入文件:

with open('resolvedworesolution.json', 'a') as f:
    f.write(str(resolvedwo))
    f.write("\n")

OR

for url in URLs:
    data = requests.get(url)
    with open('resolvedworesolution.json', 'a') as f:
        f.write(data.text)
        f.write("\n")

答案 2 :(得分:0)

写入文件时,在w模式下打开文件将删除/截断内容,然后再写入文件。

with open('resolvedworesolution.json', 'a') as f:

那应该可以解决您的问题

答案 3 :(得分:0)

您可以将信息存储在可以整体序列化的对象中:

import json
import requests

URLs = ['http://httpbin.org/ip',
'http://httpbin.org/user-agent',
'http://httpbin.org/headers']

json_list = []
for url in URLs:
    data = requests.get(url)
    resolvedwo = data.json()
    json_list.append(resolvedwo)

with open('resolvedworesolution.json', 'w+') as f:
    json.dump(json_list, f)

答案 4 :(得分:-1)

代替:

resolvedwo = data.json()

您可能想要:

resolvedwo += data.json()