Question

如果我必须减少一对值，我该如何编写lambda表达式。

testr =  [('r1', (1, 1)), ('r1', (1, 5)),('r2', (1, 1)),('r3', (1, 1))]

所需的输出

('r1', (2, 6)),('r2', (1, 1)),('r3', (1, 1))

Answer 1

按键减少它：

.reduceByKey(lambda a, b: (a[0]+b[0], a[1]+b[1]))

使用zip：

可以使任意长度元组更通用

.reduceByKey(lambda a, b: tuple(x+y for x,y in zip(a,b)))

Answer 2

it is not clear for me how reduce can use to reduce with lambda to reduce list tuples with different keys. My solution is can reduce list of tuples, but it uses function, which is perhaps too troublesome to do in pure lambda, if not impossible.

def reduce_tuple_list(tl):

    import operator as op
    import functools as fun
    import itertools as it

    # sort the list for groupby
    tl = sorted(tl,key=op.itemgetter(0))
    # this function with reduce lists with the same key
    def reduce_with_same_key(tl):
        def add_tuple(t1,t2):
            k1, tl1 = t1
            k2, tl2 = t2
            if k1 == k2:
                l1,r1 = tl1
                l2,r2 = tl2
                l = l1+l2
                r = r1+r2
                return  k1,(l,r)
            else:
                return t1,t2
        return tuple(fun.reduce(add_tuple, tl))

    # group by keys
    groups = []
    for k, g in it.groupby(tl, key=op.itemgetter(0)):
        groups.append(list(g))

    new_list = []
    # we need to add only lists whose length is greater than one
    for el in groups:
        if len(el) > 1: # reduce
            new_list.append(reduce_with_same_key(el))
        else: # single tuple without another one with the same key
            new_list.append(el[0])
    return new_list


    testr =  [('r1', (1, 1)), ('r3', (11, 71)), ('r1', (1, 5)),('r2', (1, 1)),('r3', (1, 1))]

    >>> reduce_tuple_list(testr)

    [('r1', (2, 6)), ('r2', (1, 1)), ('r3', (12, 72))]

Answer 3

你可以使用combineByKey方法

testr =  sc.parallelize((('r1', (1, 1)), ('r1', (1, 5)),('r2', (1, 1)),('r3', (1, 1))))

testr.combineByKey(lambda x:x,lambda x,y:(x[0]+y[0],x[1]+y[1]),lambda x,y:(x[0]+x[1],y[0]+y[1])).collect()

减少对 - 蟒蛇

3 个答案: