Question

我想从设置文件中动态加载列表/元组。

我需要编写一个抓取网站的抓取工具，但我希望知道找到的文件而不是网页。

我允许用户在settings.py文件中指定此类文件类型，如下所示：

# Document Types during crawling
textFiles = ['.doc', '.docx', '.log', '.msg', '.pages', '.rtf', '.txt', '.wpd', '.wps']
dataFiles = ['.csv', '.dat', '.efx', '.gbr', '.key', '.pps', '.ppt', '.pptx', '.sdf', '.tax2010', '.vcf', '.xml']
audioFiles = ['.3g2','.3gp','.asf','.asx','.avi','.flv','.mov','.mp4','.mpg','.rm','.swf','.vob','.wmv']


#What lists would you like to use ?
fileLists = ['textFiles', 'dataFiles', 'audioFiles']

我在crawler.py

中导入我的设置文件

我使用beautifulsoup模块从HTML内容和流程中查找链接，如下所示：

for item in soup.find_all("a"):
            # we dont want some of them because it is just a link to the current page or the startpage
            if item['href'] in dontWantList:
                continue

            #check if link is a file based on the fileLists from the settings
            urlpath = urlparse.urlparse(item['href']).path
            ext = os.path.splitext(urlpath)[1]
            file = False
            for list in settings.fileLists:
                if ext in settings.list:
                    file = True
                    #found file link
                    if self.verbose:
                        messenger("Found a file of type: %s" % ext, Colors.PURPLE)
                    if ext not in fileLinks:
                        fileLinks.append(item['href'])

            #Only add the link if it is not a file
            if file is not True:
                links.append(item['href'])
            else:
                #Do not add the file to the other lists
                continue

以下代码段引发错误：

 for list in settings.fileLists:
                if ext in settings.list:

显然是因为python认为settings.list是一个列表。

有没有告诉python动态查看设置文件中的列表？

Answer 1

我认为你所寻找的不是：

if ext in settings.list:

你需要

ext_list = getattr(settings, list)
if ext in ext_list:

编辑：我同意列表中的jonrsharpe，所以我在我的代码中重命名了

Python从设置文件中动态加载元组/列表

1 个答案: