通过键连接多个iteratorars

时间:2011-10-07 15:10:47

标签: python

给定:n个迭代器,以及为每个项获取项的键的函数

假设:

  • 迭代器提供按键排序的项目
  • 来自任何迭代器的键都是唯一的

我想通过键遍历它们。例如,给出以下2个列表:

[('a', {type:'x', mtime:Datetime()}), ('b', {type='y', mtime:Datetime()})]
[('b', Datetime()), ('c', Datetime())]

使用每个元组中的第一项作为键,我想得到:

(('a', {type:'x', mtime:Datetime()}), None)
(('b', {type:'y', mtime:Datetime()}), ('b', Datetime()),)
(None, ('c', Datetime()),)

所以我修改了这个方法:

def iter_join(*iterables_and_key_funcs):
    iterables_len = len(iterables_and_key_funcs)

    keys_funcs = tuple(key_func for iterable, key_func in iterables_and_key_funcs)
    iters = tuple(iterable.__iter__() for iterable, key_func in iterables_and_key_funcs)

    current_values = [None] * iterables_len
    current_keys= [None] * iterables_len
    iters_stoped = [False] * iterables_len

    def get_nexts(iters_needing_fetch):
        for i, fetch in enumerate(iters_needing_fetch):
            if fetch and not iters_stoped[i]:
                try:
                    current_values[i] = iters[i].next()
                    current_keys[i] = keys_funcs[i](current_values[i])
                except StopIteration:
                    iters_stoped[i] = True
                    current_values[i] = None
                    current_keys[i] = None

    get_nexts([True] * iterables_len)

    while not all(iters_stoped):
        min_key = min(key
                      for key, iter_stoped in zip(current_keys, iters_stoped)
                      if not iter_stoped)

        keys_equal_to_min = tuple(key == min_key for key in current_keys)
        yield tuple(value if key_eq_min else None
                    for key_eq_min, value in zip(keys_equal_to_min, current_values))

        get_nexts(keys_equal_to_min)

并测试它:

key_is_value = lambda v: v
a = (  2, 3, 4,  )
b = (1,          )
c = (          5,)
d = (1,   3,   5,)
l = list(iter_join(
        (a, key_is_value),
        (b, key_is_value),
        (c, key_is_value),
        (d, key_is_value),
    ))
import pprint; pprint.pprint(l)

输出:

[(None, 1, None, 1),
 (2, None, None, None),
 (3, None, None, 3),
 (4, None, None, None),
 (None, None, 5, 5)]

是否有现成的方法来执行此操作?我检查了itertools,但找不到任何东西。

有没有办法改进我的方法?使它更简单,更快速等。

更新:使用的解决方案

我决定通过要求迭代器产生元组(键,值)或元组(键,*值)来简化此函数的约定。以agf的答案为出发点,我提出了这个问题:

def join_items(*iterables):

    iters = tuple(iter(iterable) for iterable in iterables)
    current_items = [next(itr, None) for itr in iters]

    while True:
        try:
            key = min(item[0] for item in current_items if item != None)
        except ValueError:
            break

        yield tuple(item if item != None and item[0]==key else None
                    for item in current_items)

        for i, (item, itr) in enumerate(zip(current_items, iters)):
            if item != None and item[0] == key:
                current_items[i] = next(itr, None)


a = (      (2,), (3,), (4,),      )
b = ((1,),                        )
c = (                        (5,),)
d = ((1,),       (3,),       (5,),)
e = (                             )

import pprint; pprint.pprint(list(join_items(a, b, c, d, e)))

[(None, (1,), None, (1,), None),
 ((2,), None, None, None, None),
 ((3,), None, None, (3,), None),
 ((4,), None, None, None, None),
 (None, None, (5,), (5,), None)]

4 个答案:

答案 0 :(得分:2)

问题开头的例子与最后的例子不同。

对于第一个例子,我会这样做:

x = [('a', {}), ('b', {})]
y = [('b', {}), ('c', {})]
xd, yd = dict(x), dict(y)
combined = []
for k in sorted(set(xd.keys()+yd.keys())):
    row = []
    for d in (xd, yd):
        row.append((k, d[k]) if k in d else None)
    combined.append(tuple(row))

for row in combined:
    print row

给出

(('a', {}), None)
(('b', {}), ('b', {}))
(None, ('c', {}))

对于第二个例子

a = (  2, 3, 4,  )
b = (1,          )
c = (          5,)
d = (1,   3,   5,)

abcd = map(set, [a,b,c,d])
values = sorted(set(a+b+c+d))
print [tuple(v if v in row else None for row in abcd) for v in values]

给出

[(None, 1, None, 1),
 (2, None, None, None),
 (3, None, None, 3),
 (4, None, None, None),
 (None, None, 5, 5)]

但是你想要完成什么?也许您需要不同的数据结构。

答案 1 :(得分:1)

import itertools as it
import heapq
import pprint

def imerge(*iterables):
    '''
    http://code.activestate.com/recipes/491285-iterator-merge/
    Author: Raymond Hettinger

    Merge multiple sorted inputs into a single sorted output.

    Equivalent to:  sorted(itertools.chain(*iterables))

    >>> list(imerge([1,3,5,7], [0,2,4,8], [5,10,15,20], [], [25]))
    [0, 1, 2, 3, 4, 5, 5, 7, 8, 10, 15, 20, 25]

    '''
    heappop, siftup, _StopIteration = heapq.heappop, heapq._siftup, StopIteration

    h = []
    h_append = h.append
    for it in map(iter, iterables):
        try:
            next = it.next
            h_append([next(), next])
        except _StopIteration:
            pass
    heapq.heapify(h)

    while 1:
        try:
            while 1:
                v, next = s = h[0]      # raises IndexError when h is empty
                yield v
                s[0] = next()           # raises StopIteration when exhausted
                siftup(h, 0)            # restore heap condition
        except _StopIteration:
            heappop(h)                  # remove empty iterator
        except IndexError:
            return

a = (  2, 3, 4,  )
b = (1,          )
c = (          5,)
d = (1,   3,   5,)

def tag(iterator,val):
    for elt in iterator:
        yield elt,val

def expand(group):
    dct=dict((tag,val)for val,tag in group)
    result=[dct.get(tag,None) for tag in range(4)]
    return result

pprint.pprint(
    [ expand(group)
     for key,group in it.groupby(
          imerge(*it.imap(tag,(a,b,c,d),it.count())),
          key=lambda x:x[0]
          )])

说明:

  • 如果我们合并迭代器,生活会更容易。这可以做到 imerge
  • itertools.groupby为我们提供了所需的分组,如果我们将其提供给 来自imerge的结果。其余的只是琐碎的细节。

    pprint.pprint(
       [ list(group)
        for key,group in it.groupby(
             imerge(a,b,c,d))
         ] )
    # [[1, 1], [2], [3, 3], [4], [5, 5]]
    
  • 从上面的输出中可以清楚地看到我们需要跟踪 每个值的来源 - (该值来自ab等)。 这样我们就可以在正确的位置填充None s的输出。
  • 为此,我使用了it.imap(tag,(a,b,c,d),it.count())tag(a) 返回一个迭代器,它从a和计数器产生值 值。

    >>> list(tag(a,0))
    # [(2, 0), (3, 0), (4, 0)]
    
  • 输出现在看起来像这样:

    pprint.pprint(
       [ list(group)
        for key,group in it.groupby(
             imerge(*it.imap(tag,(a,b,c,d),it.count())),
             key=lambda x:x[0]
             )])
    
    # [[(1, 1), (1, 3)], [(2, 0)], [(3, 0), (3, 3)], [(4, 0)], [(5, 2), (5, 3)]]
    
  • 最后,我们使用expand(group)[(1, 1), (1, 3)]更改为 [None, 1, None, 1]

答案 2 :(得分:1)

a = (  2, 3, 4,  )
b = (1,          )
c = (          5,)
d = (1,   3,   5,)

iters = [iter(x) for x in (a, b, c, d)]

this = [next(i) for i in iters]

while True:
    try:
        key = min(i for i in this if i != None)
    except ValueError:
        break
    for i, val in enumerate(this):
        if val == key:
            print val,
            this[i] = next(iters[i], None)
        else:
            print None,
    print

答案 3 :(得分:0)

我意识到答案已经确定。但是我认为可以使用集合库中的defaultdict对象编写更清晰的方法。

以下是http://docs.python.org/release/2.6.7/library/collections.html

的示例
>>> s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]
>>> d = defaultdict(list)
>>> for k, v in s:
...    d[k].append(v)
>>>>d.items()
[('blue', [2, 4]), ('red', [1]), ('yellow', [1, 3])]