Question

我有一个python脚本，我用它来从网上抓取数据。然后将数据存储到MongoDB数据库中。数据采用以下格式：

{
  "id": "abcd",
  "value": 100.0,
  "timestamp": "2011-07-14 19:43:37"
}

我有很多像这样的数据。我想要做的是按小时对数据进行分组，并获得值的平均值，以及最小值和最大值。从我在mongodb docs中可以看到，聚合管道和map-reduce都可以按小时或平均值进行分组，然后我可以将其存储回数据库并重新运行聚合管道，或者对中间数据进行map reduce。

有没有办法一步完成，而不将数据存储在临时表中并运行新的迭代？

Answer 1

这可能会有所帮助：

from datetime import datetime
from itertools import groupby
from pprint import pprint

# assuming that collection of data objects is a list
datas = [
    {
        "id": "abcd",
        "value": 100.0,
        "timestamp": "2011-07-14 19:43:37"
    },
    {
        "id": "abcd",
        "value": 500.0,
        "timestamp": "2011-07-15 20:30:37"
    },
    {
        "id": "abcd",
        "value": 400.0,
        "timestamp": "2011-07-15 20:30:38"
    }
]

decorated_datas = []
# first we need to add a key with each data, that would be needed during sorting
# and that key would be date and hour
for data in datas:
    timestamp = datetime.strptime(data["timestamp"], "%Y-%m-%d %H:%M:%S") # assuming your timestamp is in this format only
    decorated_datas.append((timestamp.date(), timestamp.time().hour, data))

# then we sort the data created in the last step using the date and hour
sorted_decorated_datas = sorted(decorated_datas, key=lambda x: (x[0], x[1]))


# function for calculating statistics of a given collection of numbers
def calculate_stats(collection_of_numbers):
    maxVal = max(collection_of_numbers)
    minVal = min(collection_of_numbers)
    avgVal = sum(collection_of_numbers) / len(collection_of_numbers)
    return (maxVal, minVal, avgVal)

results = []

# then we group our sorted data by date and hour, and then we calculate
# statistics for the group and append result to our final results
for key, group_iter in groupby(sorted_decorated_datas, lambda x: (x[0], x[1])):
    group_values = [data[2]["value"] for data in group_iter]
    maxValue, minValue, avgValue = calculate_stats(group_values)

    result = {"date": key[0], "hour": key[1], "minVal":
              minValue, "maxVal": maxValue, "avgVal": avgValue}
    results.append(result)


pprint(results)

输出结果为：

[{'avgVal': 100.0,
  'date': datetime.date(2011, 7, 14),
  'hour': 19,
  'maxVal': 100.0,
  'minVal': 100.0},
 {'avgVal': 450.0,
  'date': datetime.date(2011, 7, 15),
  'hour': 20,
  'maxVal': 500.0,
  'minVal': 400.0}]

修改仔细考虑之后，我发现在字符串中表示时间戳的格式是不需要转换为datetime对象的完美候选者，并且这些时间戳字符串可以自己排序而不将它们转换为datetime对象，所以这里是更新的代码：

from itertools import groupby from pprint import pprint # assuming that collection of data objects is a list datas = [ { "id": "abcd", "value": 100.0, "timestamp": "2011-07-14 19:43:37" }, { "id": "abcd", "value": 500.0, "timestamp": "2011-07-15 20:30:37" }, { "id": "abcd", "value": 400.0, "timestamp": "2011-07-15 20:30:38" } ] def get_date_and_hour(timestamp_str): date, time = timestamp_str.split() date = tuple(map(int, date.split('-'))) hour = int(time.split(':')[0]) return tuple((*date, hour)) def calculate_stats(collection_of_numbers): maxVal = max(collection_of_numbers) minVal = min(collection_of_numbers) avgVal = sum(collection_of_numbers) / len(collection_of_numbers) return (maxVal, minVal, avgVal) results = [] sorted_datas = sorted(datas, key=lambda x: x["timestamp"]) for key, group_iter in groupby(sorted_datas, lambda x: get_date_and_hour(x["timestamp"])): group_values = [data["value"] for data in group_iter] maxValue, minValue, avgValue = calculate_stats(group_values) result = {"date": key[0:3], "hour": key[3], "minVal": minValue, "maxVal": maxValue, "avgVal": avgValue} results.append(result) pprint(results)

，输出结果为：

[{'avgVal': 100.0, 'date': (2011, 7, 14), 'hour': 19, 'maxVal': 100.0, 'minVal': 100.0}, {'avgVal': 450.0, 'date': (2011, 7, 15), 'hour': 20, 'maxVal': 500.0, 'minVal': 400.0}]

此版本比前一版本更短，更易于维护。

在mongoDB中执行多个聚合函数

1 个答案: