Question

我试图在我的代码中限制API调用。我已经找到了一个不错的python库ratelimiter==1.0.2.post0 https://pypi.python.org/pypi/ratelimiter

但是，此库只能限制本地范围内的速率。即）在函数和循环中

# Decorator
@RateLimiter(max_calls=10, period=1)
def do_something():
    pass


# Context Manager
rate_limiter = RateLimiter(max_calls=10, period=1)

for i in range(100):
    with rate_limiter:
        do_something()

因为我在不同的地方有几个函数，它们进行API调用，所以我想限制全局范围内的API调用。

例如，假设我想将API调用限制为每秒一次。并且，假设我有函数x和y，其中有两个API调用。

@rate(...)
def x():
   ...

@rate(...)
def y():
   ...

通过使用limiter修饰功能，我能够针对这两个功能限制费率。

但是，如果我按顺序执行上述两个函数，它会忽略全局范围内的API调用次数，因为它们彼此不了解。因此，y将在x执行后立即调用，而无需等待另一秒。并且，这将违反每秒一次限制。

我可以使用任何方法或库来限制python中全局的费率吗？

Answer 1

毕竟，我实现了自己的Throttler课程。通过将每个API请求代理到request方法，我们可以跟踪所有API请求。利用传递函数作为request方法参数，它还会缓存结果以减少API调用。

class TooManyRequestsError(Exception):
    def __str__(self):
        return "More than 30 requests have been made in the last five seconds."


class Throttler(object):
    cache = {}

    def __init__(self, max_rate, window, throttle_stop=False, cache_age=1800):
        # Dict of max number of requests of the API rate limit for each source
        self.max_rate = max_rate
        # Dict of duration of the API rate limit for each source
        self.window = window
        # Whether to throw an error (when True) if the limit is reached, or wait until another request
        self.throttle_stop = throttle_stop
        # The time, in seconds, for which to cache a response
        self.cache_age = cache_age
        # Initialization
        self.next_reset_at = dict()
        self.num_requests = dict()

        now = datetime.datetime.now()
        for source in self.max_rate:
            self.next_reset_at[source] = now + datetime.timedelta(seconds=self.window.get(source))
            self.num_requests[source] = 0

    def request(self, source, method, do_cache=False):
        now = datetime.datetime.now()

        # if cache exists, no need to make api call
        key = source + method.func_name
        if do_cache and key in self.cache:
            timestamp, data = self.cache.get(key)
            logging.info('{} exists in cached @ {}'.format(key, timestamp))

            if (now - timestamp).seconds < self.cache_age:
                logging.info('retrieved cache for {}'.format(key))
                return data

        # <--- MAKE API CALLS ---> #

        # reset the count if the period passed
        if now > self.next_reset_at.get(source):
            self.num_requests[source] = 0
            self.next_reset_at[source] = now + datetime.timedelta(seconds=self.window.get(source))

        # throttle request
        def halt(wait_time):
            if self.throttle_stop:
                raise TooManyRequestsError()
            else:
                # Wait the required time, plus a bit of extra padding time.
                time.sleep(wait_time + 0.1)

        # if exceed max rate, need to wait
        if self.num_requests.get(source) >= self.max_rate.get(source):
            logging.info('back off: {} until {}'.format(source, self.next_reset_at.get(source)))
            halt((self.next_reset_at.get(source) - now).seconds)

        self.num_requests[source] += 1
        response = method()  # potential exception raise

        # cache the response
        if do_cache:
            self.cache[key] = (now, response)
            logging.info('cached instance for {}, {}'.format(source, method))

        return response

Answer 2

许多API提供程序都限制了开发人员进行过多的API调用。

Python ratelimit 软件包引入了一个函数装饰器，以防止函数调用次数超过API提供者所允许的次数。

来自ratelimit进口限制

import requests
TIME_PERIOD = 900   # time period in seconds

@limits(calls=15, period=TIME_PERIOD)
def call_api(url):
    response = requests.get(url)

    if response.status_code != 200:
        raise Exception('API response: {}'.format(response.status_code))
    return response

注意：此功能将无法在15分钟的时间内进行15次以上的API调用。

Answer 3

要添加到Sunil答案中，您需要添加@sleep_and_retry装饰器，否则您的代码将在达到速率限制时中断：

@sleep_and_retry
@limits(calls=0.05, period=1)
def api_call(url, api_key):
    r = requests.get(
        url,
        headers={'X-Riot-Token': api_key}
        )
    if r.status_code != 200:
        raise Exception('API Response: {}'.format(r.status_code))
    return r

Answer 4

我遇到了同样的问题，我有很多不同的函数调用相同的API，并且我想使速率限制在全球范围内起作用。我最终要做的是创建一个启用了速率限制的空函数。

PS：：我使用了一个不同的速率限制库：https://pypi.org/project/ratelimit/

from ratelimit import limits, sleep_and_retry

# 30 calls per minute
CALLS = 30
RATE_LIMIT = 60

@sleep_and_retry
@limits(calls=CALLS, period=RATE_LIMIT)
def check_limit():
''' Empty function just to check for calls to API '''
return

然后，我只在调用API的每个函数的开头调用该函数：

def get_something_from_api(http_session, url):
    check_limit()
    response = http_session.get(url)
    return response

如果达到限制，程序将进入睡眠状态，直到（以我为例）60秒过去，然后才能正常恢复。

Answer 5

有许多精美的库将提供漂亮的装饰器和特殊的安全功能，但是以下库应与django.core.cache或任何其他具有get和set方法的缓存一起使用：

def hit_rate_limit(key, max_hits, max_hits_interval):
    '''Implement a basic rate throttler. Prevent more than max_hits occurring
    within max_hits_interval time period (seconds).'''
    # Use the django cache, but can be any object with get/set
    from django.core.cache import cache
    hit_count = cache.get(key) or 0
    logging.info("Rate Limit: %s --> %s", key, hit_count)
    if hit_count > max_hits:
        return True
    cache.set(key, hit_count + 1, max_hits_interval)
    return False

Python API速率限制 - 如何限制全局API调用

5 个答案: