Question

如何从REST API读取数据并将获取的数据插入MongoDB？我知道我们可以使用请求库读取REST API数据并获取输出。但是如何使用python将获得的输出插入MongoDB？

import requests

url='http://climatedataapi.worldbank.org/climateweb/rest/v1/country/cru/tas/year/CAN.csv'

response = requests.get(url)

if response.status_code != 200:
    print('Failed to get data:', response.status_code)
else:
    print('First 100 characters of data are')
    print(response.text[:100])

运行此代码时，我得到以下输出：

First 100 characters of data are
year,data
1901,-7.67241907119751
1902,-7.862711429595947
1903,-7.910782814025879
1904,-8.15572929382

我想插入到mongodb中，而不是进行打印。为此，我是否必须将输出存储在文件中，然后将其插入到mongodb中？如果可以，该怎么办？

Answer 1

您可以以json到requests的形式获取响应。

教程：HTTP requests and JSON parsing in Python

然后只需要将您想要的内容作为文档插入mondoDB中即可。您需要一个客户； pymongo是一个不错的选择。

教程：http://api.mongodb.com/python/current/tutorial.html

Answer 2

希望这会有所帮助，

# *-* coding: utf-8 *-*

import requests

try:
    from pymongo import MongoClient
except ImportError:
    raise ImportError('PyMongo is not installed')


class MongoDB(object):
    def __init__(self, host='localhost', port=27017, database_name=None, collection_name=None):
        try:
            self._connection = MongoClient(host=host, port=port, maxPoolSize=200)
        except Exception as error:
            raise Exception(error)
        self._database = None
        self._collection = None
        if database_name:
            self._database = self._connection[database_name]
        if collection_name:
            self._collection = self._database[collection_name]

    def insert(self, post):
        # add/append/new single record
        post_id = self._collection.insert_one(post).inserted_id
        return post_id


url = 'http://climatedataapi.worldbank.org/climateweb/rest/v1/country/cru/tas/year/CAN.csv'
response = requests.get(url)
data = response.text
if response.status_code != 200:
    print('Failed to get data:', response.status_code)
else:
    print('First 100 characters of data are')
    print(data[:100])

print('[*] Parsing response text')
data = data.split('\n')
data_list = list()
for value in data:
    if 'year,data' not in value:
        if value:
            value = value.split(',')
            data_list.append({'year': int(value[0]), 'data': float(value[1])})

print(data_list)

print('[*] Pushing data to MongoDB ')
mongo_db = MongoDB(database_name='Climate_DB', collection_name='climate_data')

for collection in data_list:
    print('[!] Inserting - ', collection)
    mongo_db.insert(collection)

输出-

First 100 characters of data are
year,data
1901,-7.67241907119751
1902,-7.862711429595947
1903,-7.910782814025879
1904,-8.15572929382
[*] Parsing response text
[{'year': 1901, 'data': -7.67241907119751}, {'year': 1902, 'data': -7.862711429595947}, {'year': 1903, 'data': -7.910782814025879}, {'year': 1904, 'data': -8.155729293823242}, {'year': 1905, 'data': -7.547311305999756}, {'year': 1906, 'data': -7.684103488922119}, {'year': 1907, 'data': -8.413553237915039}, {'year': 1908, 'data': -7.790929317474365}, {'year': 1909, 'data': -8.23930549621582}, {'year': 1910, 'data': -7.774611473083496}, {'year': 1911, 'data': -8.114446640014648}, {'year': 1912, 'data': -7.885402679443359}, {'year': 1913, 'data': -7.987940311431885}, {'year': 1914, 'data': -7.965937614440918}]
[*] Pushing data to MongoDB 
[!] Inserting -  {'year': 1901, 'data': -7.67241907119751}
[!] Inserting -  {'year': 1902, 'data': -7.862711429595947}
[!] Inserting -  {'year': 1903, 'data': -7.910782814025879}
[!] Inserting -  {'year': 1904, 'data': -8.155729293823242}
[!] Inserting -  {'year': 1905, 'data': -7.547311305999756}
[!] Inserting -  {'year': 1906, 'data': -7.684103488922119}

安装要求

pip安装pymongo

MongoDB Windows设置- https://gist.github.com/vijayanandrp/ddf72c2089015ee3f13be2b1a06bfd08

如何使用Python从REST API读取数据并将数据插入到MongoDB中？

2 个答案: