使用mapreduce的聚合框架从MongoDB中的事件数据中嵌入文档

时间：2014-08-13 18:55:03

标签： javascript mongodb mapreduce mongodb-query aggregation-framework

我有以下数据结构，并且希望将包含loc字段的文档嵌入到包含持续时间的文档中，但仅当时间戳在时间戳（ts）内减去父文档的持续时间（秒）时。 这是可能的聚合框架还是map reduce？

{
    "_id" : ObjectId("53df2a44e6583c76253c9869"),
    "deviceId" : NumberLong(1377700226807),
    "ts" : ISODate("2014-08-04T08:37:55.000Z"),
    "duration" : NumberLong(1642),
}
{
    "_id" : ObjectId("53df2a41e6583c4e243c9869"),
    "deviceId" : NumberLong(1377700226807),
    "ts" : ISODate("2014-08-04T08:37:53.000Z"),
    "loc" : {
        "lon" : 5.1101453,
        "lat" : 52.0625047
    }
}
{
    "_id" : ObjectId("53df2a3fe6583c38203c986a"),
    "deviceId" : NumberLong(1377700226807),
    "ts" : ISODate("2014-08-04T08:37:50.000Z"),
        "loc" : {
            "lon" : 5.1101297,
            "lat" : 52.0625031
        }
}
{
    "_id" : ObjectId("53df2a44e6583c76253c9869"),
    "deviceId" : NumberLong(1377700226807),
    "ts" : ISODate("2014-08-04T06:37:55.000Z"),
    "duration" : NumberLong(3600),
}
{
    "_id" : ObjectId("53df2a38e6583c03253c9869"),
    "deviceId" : NumberLong(1377700226807),
    "ts" : ISODate("2014-08-04T06:37:44.000Z"),
        "loc" : {
            "lon" : 5.1101176,
            "lat" : 52.0625171
        }
}
{
    "_id" : ObjectId("53df2a33e6583c51243c9869"),
    "deviceId" : NumberLong(1377700226807),
    "ts" : ISODate("2014-08-04T06:37:38.000Z"),
        "loc" : {
            "lon" : 5.1101409,
            "lat" : 52.0625818
        }
}
{
    "_id" : ObjectId("53df2a2de6583c38203c9869"),
    "deviceId" : NumberLong(1377700226807),
    "ts" : ISODate("2014-08-04T06:37:32.000Z"),
        "loc" : {
            "lon" : 5.1099513,
            "lat" : 52.0624157
        }
}

这是所需的格式 

{
    "_id" : ObjectId("53df2a44e6583c76253c9869"),
    "deviceId" : NumberLong(1377700226807),
    "ts" : ISODate("2014-08-04T08:37:55.000Z"),
    "duration" : NumberLong(1642),
    "data" : [
       {
          "ts" : ISODate("2014-08-04T08:37:53.000Z"),
          "loc" : {
             "lon" : 5.1101453,
             "lat" : 52.0625047
          }
       },
       {
          "ts" : ISODate("2014-08-04T08:37:50.000Z"),
          "loc" : {
            "lon" : 5.1101297,
            "lat" : 52.0625031
           }
       }
    ]
}

2 个答案:

答案 0 :(得分：1)

聚合框架无法保留＆＃34;处理管道时跨文档的信息。那种＆＃34;父母/孩子＆＃34;您正在描述的关系，通过与父母的比较来确定该元素的位置＆＃34;因此，无法在文件中直接指明的文件。

然而，mapReduce方法可以访问＆＃34; global＆＃34;范围变量。这可以允许识别＆＃34;然后可以将父细节存储在变量中，以便根据需要与可能的子项进行比较。

db.collection.mapReduce(
  function() {
    lastValue.data = [];

    if ( lastId == null || this.hasOwnProperty("duration") ) {
      lastId = this._id;
      lastValue.deviceId = this.deviceId;
      lastValue.ts = this.ts;
      lastValue.duration = this.duration;
    }

    if (
        ( this.hasOwnProperty("loc") ) &&
        (
          ( lastValue.ts.valueOf() - ( lastValue.duration * 1000 ) ) <
          this.ts.valueOf()
        )
      )
      lastValue.data.push({
        "ts": this.ts,
        "loc": this.loc
      });


    emit ( lastId, lastValue );

  },
  function (key,values) {

    var reduced = {};

    values.forEach(function(value) {
      if ( !reduced.hasOwnProperty("deviceId") ) {
        reduced.deviceId = value.deviceId;
        reduced.ts = value.ts;
        reduced.duration = value.duration;
        reduced.data = [];
      }

      value.data.forEach(function(dat) {
        reduced.data.push(dat);
      });

    });

    return reduced;

  },
  { 
      "sort": { "ts": -1 },
      "scope": { "lastId": null, "lastValue": {} }, 
      "out": { "inline": 1 }
  }
)

所以基本上这是＆＃34;存储＆＃34; ＆＃34;键＆＃34;作为＆＃34; lastId＆＃34;使用＆＃34;范围＆＃34; mapReduce可用的声明。清除＆＃34;父母＆＃34;文档包含持续时间，因此可以在此处使用。

＆＃34;数据＆＃34;元素需要作为数组发出。这是由文档中描述的requirement of the "reduce" function引起的。关键是输入必须与预期输入的格式相同。 ＆＃34;减少＆＃34;每个＆＃34;键＆＃34;。

可以多次调用函数

＆＃34;映射器中的条件＆＃34;可以在＆＃34; reduce＆＃34;之前限制此数组中存在的值。虽然调用了函数。这也避免了依赖于添加＆＃34; finalize＆＃34;如果只有一个＆＃34;键＆＃34;发出，＆＃34;减少＆＃34;函数不会触摸此项目。因此，预先进行过滤的工作量会减少。

＆＃34;减速机＆＃34;然后现在降级为＆＃34;合并＆＃34;将结果组合在＆＃34;数据＆＃34;有资格列入＆＃34;儿童＆＃34;。

自然地，排序顺序是＆＃34;降序＆＃34;时间戳或＆＃34; ts＆＃34;值。这将是索引的一个好点，并且顺序与文档的处理方式一致，以便进行比较，因此每个＆＃34;断点＆＃34;在检测到新父项的地方有效。

输出当然是mapReduce样式。所以到达你的'＃34;确切＆＃34;期望的输出需要更多的后期处理，但正如您所看到的那样基本上是结果：

    "results" : [
        {
            "_id" : ObjectId("53f15b8beb75bdce84f914a1"),
            "value" : {
                "deviceId" : NumberLong("1377700226807"),
                "ts" : ISODate("2014-08-04T08:37:55Z"),
                "duration" : NumberLong(1642),
                "data" : [
                    {
                        "ts" : ISODate("2014-08-04T08:37:53Z"),
                        "loc" : {
                            "lon" : 5.1101453,
                            "lat" : 52.0625047
                        }
                    },
                    {
                        "ts" : ISODate("2014-08-04T08:37:50Z"),
                        "loc" : {
                            "lon" : 5.1101297,
                            "lat" : 52.0625031
                        }
                    }
                ]
             }
        },
        {
            "_id" : ObjectId("53f15b8beb75bdce84f914a4"),
            "value" : {
                "deviceId" : NumberLong("1377700226807"),
                "ts" : ISODate("2014-08-04T06:37:55Z"),
                "duration" : NumberLong(3600),
                "data" : [
                    {
                        "ts" : ISODate("2014-08-04T06:37:44Z"),
                        "loc" : {
                            "lon" : 5.1101176,
                            "lat" : 52.0625171
                        }
                    },
                    {
                        "ts" : ISODate("2014-08-04T06:37:38Z"),
                        "loc" : {
                            "lon" : 5.1101409,
                            "lat" : 52.0625818

                    },
                    {
                        "ts" : ISODate("2014-08-04T06:37:32Z"),
                        "loc" : {
                            "lon" : 5.1099513,
                            "lat" : 52.0624157
                        }
                    }
                ]
            }
        }
    ]

请注意，此处给出的输出为不同的_id值，因为样本包含＆＃34;重复＆＃34; _id的值，当然不允许在主键上使用。

答案 1 :(得分：1)

使用聚合框架并不容易，但可以使用MapReduce完成。

假设数据是收集的属性（即没有遗漏＆＃34;乘坐＆＃34;文件的持续时间为＆＃34; loc＆＃34;值的文件）你可以这样做：

map=function () {
   var startTime;
   if (this.hasOwnProperty("duration"))
       startTime=this.ts-this.duration*1000;
   else
       startTime=this.ts;
   emit(this.deviceId, {startTs:new Date(startTime), endTs:this.ts, loc:this.loc, duration:this.duration});
}

Map以标准化格式输出内容，每个deviceId将它们全部减少为一个数组。

reduce=function (key,values) {
   var result = { vals : [ ] };
   values.forEach(function(v) {
       result.vals.push(v);
   })
   return result;
}

所有实际处理（每个deviceId的分组）都发生在finalize函数中，该函数为每个deviceId获取一个数组，并对其进行排序，并将其分组到您期望的文档中。

finalize=function (key, value) {
    var lastI=-1;
    var result = {rides: [ ] };
    var ride = { };
    value.vals.sort(function(a,b) { return a.startTs.getTime() - b.startTs.getTime(); } );

    for (i=0; i<value.vals.length; i++) {
        if (value.vals[i].loc == null ) {
           if (ride.hasOwnProperty("locations")) {
               result.rides.push(ride);
               ride={};
           }
           ride["start"]=value.vals[i].startTs;
           ride["end"]=value.vals[i].endTs;
           ride["duration"]=value.vals[i].duration;
           ride["locations"]=[];
           lastI=i;
        } else {
           ride.locations.push({ loc: value.vals[i].loc, ts: value.vals[i].endTs});
        }
    }
    result.rides.push(ride);
    return result;
}

我在测试数据中添加了几个deviceIds：

db.rides.find({},{_id:0})
{ "deviceId" : NumberLong("1377700226807"), "ts" : ISODate("2014-08-04T06:37:32Z"), "loc" : { "lon" : 5.1099513, "lat" : 52.0624157 } }
{ "deviceId" : NumberLong("1377700226910"), "ts" : ISODate("2014-08-04T06:37:32Z"), "loc" : { "lon" : 5.1099513, "lat" : 52.0624157 } }
{ "deviceId" : NumberLong("1377700226807"), "ts" : ISODate("2014-08-04T06:37:38Z"), "loc" : { "lon" : 5.1101409, "lat" : 52.0625818 } }
{ "deviceId" : NumberLong("1377700226910"), "ts" : ISODate("2014-08-04T06:37:38Z"), "loc" : { "lon" : 5.1101409, "lat" : 52.0625818 } }
{ "deviceId" : NumberLong("1377700226807"), "ts" : ISODate("2014-08-04T06:37:44Z"), "loc" : { "lon" : 5.1101176, "lat" : 52.0625171 } }
{ "deviceId" : NumberLong("1377700226910"), "ts" : ISODate("2014-08-04T06:37:44Z"), "loc" : { "lon" : 5.1101176, "lat" : 52.0625171 } }
{ "deviceId" : NumberLong("1377700226807"), "ts" : ISODate("2014-08-04T06:37:55Z"), "duration" : NumberLong(3600) }
{ "deviceId" : NumberLong("1377700226910"), "ts" : ISODate("2014-08-04T06:37:55Z"), "duration" : NumberLong(3600) }
{ "deviceId" : NumberLong("1377700226807"), "ts" : ISODate("2014-08-04T08:37:50Z"), "loc" : { "lon" : 5.1101297, "lat" : 52.0625031 } }
{ "deviceId" : NumberLong("1377700226908"), "ts" : ISODate("2014-08-04T08:37:50Z"), "loc" : { "lon" : 5.1101297, "lat" : 52.0625031 } }
{ "deviceId" : NumberLong("1377700226807"), "ts" : ISODate("2014-08-04T08:37:53Z"), "loc" : { "lon" : 5.1101453, "lat" : 52.0625047 } }
{ "deviceId" : NumberLong("1377700226908"), "ts" : ISODate("2014-08-04T08:37:53Z"), "loc" : { "lon" : 5.1101453, "lat" : 52.0625047 } }
{ "deviceId" : NumberLong("1377700226807"), "ts" : ISODate("2014-08-04T08:37:55Z"), "duration" : NumberLong(1642) }
{ "deviceId" : NumberLong("1377700226908"), "ts" : ISODate("2014-08-04T08:37:55Z"), "duration" : NumberLong(1642) }

并通过MR

运行 
db.rides.mapReduce(map, reduce, {out:"newrides", finalize:finalize})
{
        "result" : "frides",
        "timeMillis" : 47,
        "counts" : {
                "input" : 14,
                "emit" : 14,
                "reduce" : 3,
                "output" : 3
        },
        "ok" : 1
}

结果是：

db.newrides.find().pretty()
{
        "_id" : NumberLong("1377700226807"),
        "value" : {
                "rides" : [
                        {
                                "start" : ISODate("2014-08-04T05:37:55Z"),
                                "end" : ISODate("2014-08-04T06:37:55Z"),
                                "duration" : NumberLong(3600),
                                "locations" : [
                                        {
                                                "loc" : {
                                                        "lon" : 5.1099513,
                                                        "lat" : 52.0624157
                                                },
                                                "ts" : ISODate("2014-08-04T06:37:32Z")
                                        },
                                        {
                                                "loc" : {
                                                        "lon" : 5.1101409,
                                                        "lat" : 52.0625818
                                                },
                                                "ts" : ISODate("2014-08-04T06:37:38Z")
                                        },
                                        {
                                                "loc" : {
                                                        "lon" : 5.1101176,
                                                        "lat" : 52.0625171
                                                },
                                                "ts" : ISODate("2014-08-04T06:37:44Z")
                                        }
                                ]
                        },
                        {
                                "start" : ISODate("2014-08-04T08:10:33Z"),
                                "end" : ISODate("2014-08-04T08:37:55Z"),
                                "duration" : NumberLong(1642),
                                "locations" : [
                                        {
                                                "loc" : {
                                                        "lon" : 5.1101297,
                                                        "lat" : 52.0625031
                                                },
                                                "ts" : ISODate("2014-08-04T08:37:50Z")
                                        },
                                        {
                                                "loc" : {
                                                        "lon" : 5.1101453,
                                                        "lat" : 52.0625047
                                                },
                                                "ts" : ISODate("2014-08-04T08:37:53Z")
                                        }
                                ]
                        }
                ]
        }
}
{
        "_id" : NumberLong("1377700226908"),
        "value" : {
                "rides" : [
                        {
                                "start" : ISODate("2014-08-04T08:10:33Z"),
                                "end" : ISODate("2014-08-04T08:37:55Z"),
                                "duration" : NumberLong(1642),
                                "locations" : [
                                        {
                                                "loc" : {
                                                        "lon" : 5.1101297,
                                                        "lat" : 52.0625031
                                                },
                                                "ts" : ISODate("2014-08-04T08:37:50Z")
                                        },
                                        {
                                                "loc" : {
                                                        "lon" : 5.1101453,
                                                        "lat" : 52.0625047
                                                },
                                                "ts" : ISODate("2014-08-04T08:37:53Z")
                                        }
                                ]
                        }
                ]
        }
}
{
        "_id" : NumberLong("1377700226910"),
        "value" : {
                "rides" : [
                        {
                                "start" : ISODate("2014-08-04T05:37:55Z"),
                                "end" : ISODate("2014-08-04T06:37:55Z"),
                                "duration" : NumberLong(3600),
                                "locations" : [
                                        {
                                                "loc" : {
                                                        "lon" : 5.1099513,
                                                        "lat" : 52.0624157
                                                },
                                                "ts" : ISODate("2014-08-04T06:37:32Z")
                                        },
                                        {
                                                "loc" : {
                                                        "lon" : 5.1101409,
                                                        "lat" : 52.0625818
                                                },
                                                "ts" : ISODate("2014-08-04T06:37:38Z")
                                        },
                                        {
                                                "loc" : {
                                                        "lon" : 5.1101176,
                                                        "lat" : 52.0625171
                                                },
                                                "ts" : ISODate("2014-08-04T06:37:44Z")
                                        }
                                ]
                        }
                ]
        }
}
相关问题
最新问题