构建大型列表的Python Discord.py程序的内存不足,并且在PythonAnywhere上崩溃

时间:2018-12-01 23:05:06

标签: python python-3.x python-asyncio discord.py

我为Discord服务器创建了一个机器人,该机器人针对给定的subreddit进入Reddit API,并根据您输入的subreddit在Discord聊天中发布当天的前10名结果。它不理会自己的帖子,实际上只发布图片和GIF。 Discord消息命令如下所示:=get funny awww news,当每个子reddit从Reddit API获取结果时,将其发布。这项工作没有问题。我知道该漫游器可以访问API并发布到不和谐的功能。

这是我需要解决的问题:

我添加了另一个命令=getshuffled,该命令将所有来自subreddit的结果放入一个较大的列表中,然后在发布之前将其洗牌。最多约50个subreddits的要求,这确实非常有效。

由于结果列表如此之大,从100多个subreddits中获得了1000多个结果,因此该机器人在遇到非常大的请求时崩溃了。现在,我想我知道问题出在:该机器人托管在PythonAnywhere上,在那里我只有3GB的RAM。现在,在我的脑海中,即使列表很大,它也不应该占用太多内存。没门。所以,我想我没有清除内存吗?

我敢肯定,从Reddit提取数据并创建大结果列表会引起很多麻烦,然后当它尝试改组并将结果发布到下一个函数中时,它将耗尽内存,并且PythonAnywhere会杀死过程。

这就是我遇到的问题:我不太确定python如何管理函数之间的内存,即使我做对了也不知道。我认为这很简单,例如我如何调用函数或某些东西,或者我不了解Asyncio的工作原理。

这是PythonAnywhere关于RAM的内容:

由于系统RAM是PythonAnywhere服务器上最稀缺的资源之一,因此我们将您的进程的最大内存大小限制为3GB。这是每个进程的限制,而不是系统范围的限制,因此,如果您有更大的内存需求,则可以通过运行多个较小的进程来执行所需的处理。 如果一个进程超过内存限制,它将被杀死。

在这方面的任何帮助将不胜感激!

(此外,如果代码中有任何让您摇头的东西,请告诉我,我确定我犯了一些错误的操作错误!谢谢大家)

编辑:Python版本为3.6,Discord.py版本为0.16.12

编辑2:我检查了服务器日志:这是运行大量结果时抛出的错误(这是在main_post函数上抛出的,我已经注释了该行):

ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Client._run_event() running at /home/GageBrk/.local/lib/python3.6/site-packages/discord/client.py:307> wait_for=<Future pending cb=[BaseSelectorEventLoop._sock_connect_done(13)(), <TaskWakeupMethWrapper object at 0x7f0aa5bbaa38>()]>>

代码:

import logging
import socket
import sys
import random
import os

from redditBot_auth import reddit

import discord
import asyncio
from discord.ext.commands import Bot
#from discord.ext import commands
import platform

@client.event
async def on_ready():    
    return await client.change_presence(game=discord.Game(name='Getting The Dank Memes')) 


def is_number(s):
    try:
        int(s)
        return True
    except:
        pass

def show_title(s):
    try:
        if s == 'TITLE':
            return True
        if s == 'TITLES':
            return True
        if s == 'title':
            return True
        if s == 'titles':
            return True
    except:
        pass


#This gets results for each subreddit and puts into nested list with format 
#[[title1, URL1], [title2, URL2], [title3, URL3]]
#It then passes that results list to the function that posts to Discord
async def main_loop(*args, shuffled=False): 

    q=10

    #This takes a integer value argument from the input string.
    #It sets the number variable,
    #Then deletes the number from the arguments list.
    #same with the title variable
    title = False
    for item in args: 
        if is_number(item):
            q = item
            q = int(q)
            if q > 15:
                q=15
            args = [x for x in args if not is_number(x)]

        if show_title(item):
            title = True
            args = [x for x in args if not show_title(x)]

    number_of_posts = q * len(args)
    results=[]
    #results = [[] for x in range(number_of_posts)] #create nested lists for each post. This is ALL the links that will be posted.

    TESTING = False #If this is turned to True, the subreddit of each post will be posted. Will use defined list of results
    NoGrabResults = False

    i = 0

    await client.say('*Posting ' + str(number_of_posts) + ' posts from ' + str(len(args))+' subreddits*')
    #This pulls the data and creates a list of links for the bot to post

    if NoGrabResults == False: #This is for testing, ignore
        for item in args:
            try:

                #subreddit_results = [[] for x in range(q)] #create nested lists for each subreddit.
                                                            #This allows for duplicate deletion within the subreddit results,
                                                            #rather than going over the entire 'results' list.
                subreddit_results=[]

                e = 0 #counter for the subreddit_results list.

                #await client.say('<'+item+'>')
                subreddit = reddit.subreddit(item)
                Day = subreddit.top('day', limit= q*2)
                Week = subreddit.top('week', limit = q*2)
                Month = subreddit.top('month', limit = q*2)
                Year = subreddit.top('year', limit = q*2)
                AllTime = subreddit.top('all', limit = q*2)

                print(item)

                for submission in Day:

                    post = []
                    if len(subreddit_results) < q :
                        if submission.is_self is False:
                            if '/v.redd.it/' not in submission.url:
                                #print(submission.url)
                                if '.gif' or 'imgur.com' or 'gfycat' in submission.url:
                                    if submission.url not in subreddit_results:
                                        post.append(submission.title)
                                        post.append(submission.url)
                                        #post.append(item)
                                        subreddit_results.append(post)

                #print('pulled posts from Daily')

                if len(subreddit_results) < q :
                    #print('getting posts from Weekly')
                    for submission in Week:
                        post = []
                        if len(subreddit_results) < q :
                            if submission.is_self is False:
                                if '/v.redd.it/' not in submission.url:
                                    #print(submission.url)
                                    if '.gif' or 'imgur.com' or 'gfycat' in submission.url:
                                        if submission.url not in subreddit_results:
                                            post.append(submission.title)
                                            post.append(submission.url)
                                            #post.append(item)
                                            subreddit_results.append(post)


                if len(subreddit_results) < q :
                    #print('getting posts from Monthly')
                    for submission in Month:
                        post = []
                        if len(subreddit_results) < q :
                            if submission.is_self is False:
                                if '/v.redd.it/' not in submission.url:
                                    #print(submission.url)
                                    if '.gif' or 'imgur.com' or 'gfycat' in submission.url:
                                        if submission.url not in subreddit_results:
                                            post.append(submission.title)
                                            post.append(submission.url)
                                            #post.append(item)
                                            subreddit_results.append(post)


                if len(subreddit_results) < q :
                    #print('getting posts from Yearly')
                    for submission in Year:
                        post = []
                        if len(subreddit_results) < q :
                            if submission.is_self is False:
                                if '/v.redd.it/' not in submission.url:
                                    if '.gif' or 'imgur.com' or 'gfycat' in submission.url:
                                        if submission.url not in subreddit_results:
                                            post.append(submission.title)
                                            post.append(submission.url)
                                            #post.append(item)
                                            subreddit_results.append(post)


                if len(subreddit_results) < q :
                    #print('getting posts from All Time')
                    for submission in AllTime:
                        post = []
                        if len(subreddit_results) < q :
                            if submission.is_self is False:
                                if '/v.redd.it/' not in submission.url:
                                    if '.gif' or 'imgur.com' or 'gfycat' in submission.url:
                                        if submission.url not in subreddit_results:
                                            post.append(submission.title)
                                            post.append(submission.url)
                                            #post.append(item)
                                            subreddit_results.append(post)


                #print (subreddit_results)

                # If they don't want shuffled results, it will post results
                # to Discord as it gets them, instead of creating the nested list
                if shuffled == False:
                    await client.say('<'+item+'>')
                    for link in subreddit_results:
                        if title==True:
                            await client.say(link[0]) #title
                        await client.say(link[1]) #post url 1
                        if TESTING == True:
                            await client.say(link[2]) #subreddit
                        if title==True:
                            await client.say("_") #spacer to seperate title from post above
                else:
                    for link in subreddit_results:
                        results.append(link)






            except Exception as e:
                if 'Redirect to /subreddits/search' or '404' in str(e):
                   await client.say('*'+item+' failed...* '+'`Subreddit Does Not Exist`')
                if '403' in str(e):
                   await client.say('*'+item+' failed...* '+'`Access Denied`')
                print(str(e) + ' --> ' + item)
                pass
        print ('results loaded')
        await main_post(results, shuffled, title, TESTING)

    else:
        from Test_args import LargeResults as results
    #print(results)
        await main_post(results, shuffled, title, TESTING)

#this shuffles the posts and posts to Discord.
async def main_post(results, shuffled, title, TESTING):
    try:
        if shuffled == True:
            print('____SHUFFLED___')
            random.shuffle(results)
            random.shuffle(results)
            random.shuffle(results)

        #This posts the links in the 'results' list to Discord

        for post in results:

            try:
                #                                 THIS IS WHERE THE PROGRAM IS FAILING!!
                if title==True:
                    await client.say(post[0]) #title

                await client.say(post[1]) #post url 

                if TESTING == True:
                    await client.say(post[2]) #subreddit

                if title==True:
                    await client.say("_") #spacer to separate title from post above
            except Exception as e:
                print(e)
                pass

        await client.say('ALL DONE! ! !')

    except Exception as e:
        print (e)
        pass
        await client.say('`' +str(e) +'`')

@client.command()
async def get(*args, brief="say '=get' followed by a list of subreddits", description="To get the 10 Top posts from a subreddit, say '=get' followed by a list of subreddits:\n'=get funny news pubg'\n would get the top 10 posts for today for each subreddit and post to the chat."):
    #sr = '+'.join(args)
    await main_loop(*args)

#THIS POSTS THE POSTS RANDOMLY   
@client.command()
async def getshuffled(*args, brief="say '=getshuffled' followed by a list of subreddits", description="Does the same thing as =get, but grabs ALL of the posts and shuffles them, before posting."):

    await main_loop(*args, shuffled=True)


client.run('my ID')

0 个答案:

没有答案