Twitter挖掘问题与速率限制

Question

我一直在研究项目，包括从Twitter获取朋友和关注者信息。我已经编写了一个适用于样本帐户的代码，例如我的拥有1k粉丝或朋友的代码。但是我在为拥有1220万粉丝的唐纳德特朗普账户运营它时遇到了麻烦。在我的程序中查询了一段时间后，超过了速率限制，我等了15分钟然后再次请求相同。此过程适用于大约1百万用户信息，但在此之后它会卡住或有时会出现URL错误。请说明处理这些大数据时可能出现的问题以及解决方案。请查找以下相同的代码段 -

def get_info_by_screen_name(t, screen_names):

    sn_to_info = {}
    while len(screen_names) > 0:

        # Process 100 ids at a time...

        screen_names_str = ','.join([str(sn) for sn in screen_names[:100]])
        screen_names = screen_names[100:]

        response = make_twitter_request(t,
                                        getattr(getattr(t, "users"), "lookup"),
                                        screen_name=screen_names_str)

        if response is None:
            break

        if type(response) is dict:  # Handle Twitter API quirk
            response = [response]

        for user_info in response:
            sn_to_info[user_info['screen_name']] = user_info

    return sn_to_info


    t = oauth_login()

    friends_ids, followers_ids = get_friends_followers_ids(t,
                                                           screen_name="realDonaldTrump",
                                                           friends_limit=100,
                                                           followers_limit=100)

    # Basic usage...


    print friends_ids
    print followers_ids
    # Basic usage...
    info = {}
    data_test = {}
    info.update(get_info_by_id(t, friends_ids))