识别等效的varargs函数调用Python中的memoization

Question

我正在使用以下装饰器的变体进行记忆（找到here）：

# note that this decorator ignores **kwargs
def memoize(obj):
    cache = obj.cache = {}

    @functools.wraps(obj)
    def memoizer(*args, **kwargs):
        if args not in cache:
            cache[args] = obj(*args, **kwargs)
        return cache[args]
    return memoizer

我想知道，有没有一种合理的方法可以基于args和kwargs进行memoize，特别是在两个函数调用指定的情况下，参数通过不同的位置和关键字分配，但具有完全相同的参数？

Answer 1

如果您使用参数始终作为定位或始终作为关键字，Thorsten解决方案可以正常工作。但是，如果你想考虑给参数赋予相同值的相等调用，而不依赖于参数的传递方式，那么你必须做一些更复杂的事情：

import inspect


def make_key_maker(func):
    args_spec = inspect.getargspec(func)

    def key_maker(*args, **kwargs):
        left_args = args_spec.args[len(args):]
        num_defaults = len(args_spec.defaults or ())
        defaults_names = args_spec.args[-num_defaults:]

        if not set(left_args).symmetric_difference(kwargs).issubset(defaults_names):
            # We got an error in the function call. Let's simply trigger it
            func(*args, **kwargs)

        start = 0
        key = []
        for arg, arg_name in zip(args, args_spec.args):
            key.append(arg)
            if arg_name in defaults_names:
                start += 1

        for left_arg in left_args:
            try:
                key.append(kwargs[left_arg])
            except KeyError:
                key.append(args_spec.defaults[start])

            # Increase index if we used a default, or if the argument was provided
            if left_arg in defaults_names:
                start += 1
        return tuple(key)

    return key_maker

上述函数尝试将关键字参数（和默认值）映射到位置，并将结果元组用作键。我测试了一下，它似乎在大多数情况下都能正常工作。 当目标函数也使用**kwargs参数时，它会失败。

>>> def my_function(a,b,c,d,e=True,f="something"): pass
... 
>>> key_maker = make_key_maker(my_function)
>>> 
>>> key_maker(1,2,3,4)
(1, 2, 3, 4, True, 'something')
>>> key_maker(1,2,3,4, e=True)               # same as before
(1, 2, 3, 4, True, 'something')
>>> key_maker(1,2,3,4, True)                 # same as before
(1, 2, 3, 4, True, 'something')
>>> key_maker(1,2,3,4, True, f="something")  # same as before
(1, 2, 3, 4, True, 'something')
>>> key_maker(1,2,3,4, True, "something")    # same as before
(1, 2, 3, 4, True, 'something')
>>> key_maker(1,2,3,d=4)                     # same as before
(1, 2, 3, 4, True, 'something')
>>> key_maker(1,2,3,d=4, f="something")      # same as before
(1, 2, 3, 4, True, 'something')

Answer 2

import inspect
def memoize(obj):
    cache = obj.cache = {}
    @functools.wraps(obj)
    def memoizer(*args, **kwargs):
        kwargs.update(dict(zip(inspect.getargspec(obj).args, args)))
        key = tuple(kwargs.get(k, None) for k in inspect.getargspec(obj).args)
        if key not in cache:
            cache[key] = obj(**kwargs)
        return cache[key]
    return memoizer

Answer 3

通常，无法推断两个调用具有相同的参数含义。考虑一下这些电话

func(foo=1)
func(1)
func(bar=1)

这些（如果有的话）中的哪一个是等价的，取决于位置参数是否被称为foo或bar：如果参数被称为foo，则第一个调用与第二个调用匹配，等等。但是，位置参数也可能具有完全不同的名字。

IOW，你需要反思要调用的函数，反过来，这可能是不可能的（例如，如果它是用C实现的，或者它本身就是一个只处理* args，** kwargs的包装器）。

如果你想走反射路线，像ndpu的反应就是一个好的开始。

Answer 4

您必须找到一种从args和kwargs构建密钥的好方法。也许试试这个：

import functools
from collections import OrderedDict

# note that this decorator ignores **kwargs
def memoize(obj):
    def make_key(args, kwargs):
        ordered_kwargs = OrderedDict(kwargs)
        parameters = tuple([args, 
                            tuple(ordered_kwargs.keys()), 
                            tuple(ordered_kwargs.values())])
        return parameters
    cache = obj.cache = {}

    @functools.wraps(obj)
    def memoizer(*args, **kwargs):
        key = make_key(args, kwargs)
        if key not in cache:
            cache[key] = obj(*args, **kwargs)
            print "Not using cached result for key %s" % str(key)
        else:
            print "Using cached result for key %s" % str(key)
        return cache[key]
    return memoizer

@memoize
def calculate_sum(*args, **kwargs):
    return sum(args)

calculate_sum(4,7,9,2,flag=0)
calculate_sum(4,7,9,3)
calculate_sum(4,7,9,2,flag=1)
calculate_sum(4,7,9,2,flag=0)

我将一些打印语句放入memoizer，只是为了证明它有效。输出是：

Not using cached result for key ((4, 7, 9, 2), ('flag',), (0,))
Not using cached result for key ((4, 7, 9, 3), (), ())
Not using cached result for key ((4, 7, 9, 2), ('flag',), (1,))
Using cached result for key ((4, 7, 9, 2), ('flag',), (0,))

我确信我没有处理所有角落案件，特别是如果以kwargs（甚至是args）传入的值不可清除。但也许它可以作为一个很好的起点。

Answer 5

此解决方案使用inspect模块提取位置和关键字参数的参数名称。然后对名称：值对的有序元组执行memoization查找。它可以容忍作为位置和关键字参数传递的参数。如果存在多余的位置参数，则它们按照它们出现在单独元组中的顺序存储。

This uses Michele Simionato's decorator package以确保保留功能签名。因为它会检查被记忆的函数的argspec，所以如果使用不保留argspec的装饰器实现进行组合，它将会失败。

from decorator import decorator as robust_decorator

def argument_signature(function,*args,**kwargs):
    '''
    Convert the function arguments and values to a unique set. 
    Throws ValueError if the provided arguments cannot match argspec.
    '''
    named_store = {} # map from parameter names to values 
    named,vargname,kwargname,defaults = inspect.getargspec(function)
    available = zip(named,args)
    nargs     = len(available)
    ndefault  = len(defaults) if not defaults is None else 0
    nnamed    = len(named)
    # All positional arguments must be filled
    nmandatory = nnamed - ndefault
    if nargs<nmandatory: raise ValueError('Not enough positional arguments')
    # Assign available positional arguments to names    
    for k,v in available:
        if k in named_store: raise ValueError('Duplicate argument',k)
        named_store[k] = v
    # If not all arguments are provided, check **kwargs and defaults
    ndefaulted   = max(0,nnamed - nargs)
    default_map = dict(zip(named[-ndefault:],defaults)) if ndefault>0 else {}
    if ndefaulted>0:
        for k in named[-ndefaulted:]:
            if k in named_store: raise ValueError('Duplicate argument',k)
            named_store[k] = kwargs[k] if k in kwargs else default_map[k]
            if k in kwargs: del kwargs[k]
    # Store excess positional arguments in *vargs if possible
    vargs = None
    if len(args)>nnamed:
        if vargname is None:
            raise ValueError('Excess positional arguments, but function does not accept *vargs!')
        vargs = args[nnamed:]
    # Store excess keyword arguments if the function accepts **kwargs
    if len(kwargs):
        if kwargname is None:
            raise ValueError("Excess keyword arguments, but function does not accept **kwargs!")
        for k in kwargs:
            if k in named_store: raise ValueError('Duplicate argument',k)
            named_store[k] = kwargs[k]
    # Construct a tuple reflecting argument signature
    keys  = sorted(named_store.keys())
    vals  = tuple(named_store[k] for k in keys)
    named = tuple(zip(keys,vals))
    argument_signature = (named,vargs)
    return argument_signature

def print_signature(sig):
    '''Formats the argument signature for printing.'''
    named, vargs = sig
    result = ','.join(['%s=%s'%(k,v) for (k,v) in named])
    if not vargs is None: result += '; ' + ','.join(map(str,vargs))
    return result

def vararg_memoize(f):
    '''Memoization decorator'''
    cache = {}
    @robust_decorator
    def decorated(f,*args,**kwargs):
        sig = argument_signature(f,*args,**kwargs)
        if not sig in cache:  cache[sig] = f(*args,**kwargs)
        else: print('found cached',f.func_name,print_signature(sig))
        return cache[sig]
    return decorated(f)

if __name__=="__main__":
    print("Running example and testing code")

    def example_function(a,b,c=1,d=('ok',),*vargs,**kw):
        ''' This docstring should be preserved by the decorator '''
        e,f = vargs if (len(vargs)==2) else (None,None)
        g = kw['k'] if 'k' in kw else None
        print(a,b,c,d,e,f,g)

    f = example_function
    g = vararg_memoize(example_function)

    for fn in [f,g]:
        print('Testing',fn.__name__)
        fn('a','b','c','d')
        fn('a','b','c','d','e','f')
        fn('a','b',c='c',d='d')
        fn('a','b',**{'c':'c','d':'d'})
        fn('a','b',*['c','d'])
        fn('a','b',d='d',*['c'])
        fn('a','b',*['c'],**{'d':'d'})
        fn('a','b','c','d','e','f')