Question

Python beginner here. Let's say I have three methods for scraping websites. Let's call them scrape_site_a, scrape_site_b, and scrape_site_c. I want to run each of these but I'd like to define them in such a way that I can call them dynamically without calling each by name. Ideally I'd like to just load all modules in a directory and call the same method on each of them. My attempt so far is the following:

site_a.py

def scrape():
    # scrape the site

site_b.py

def scrape():
    # scrape the site

site_c.py

def scrape():
    # scrape the site

I have the __init__.py setup such that I can do the following:

scrape.py

from sites import *

site_a.scrape()
site_b.scrape()
site_c.scrape()

I would like to do something like:

for site in sites:
    site.scrape()

I realize that there is a fundamental programming concept I'm not understanding here and I have two questions:

Is there a way to do this using the approach I'm taking?
Is there a better approach? Why?

Answer 1

以下内容扫描给定目录，加载其中的每个.py文件，并调用模块的scrape方法（如果存在）。

from os import listdir
from os.path import join

scraper_dir = "./scrapers"

for scraper_name in listdir(scraper_dir):
    if scraper_name.endswith(".py"):
        with open(join(scraper_dir, scraper_name)) as scraper_file:
            scraper_globals = {}              # this will hold scraper's globals
            scraper_module = exec(scraper_file.read(), scraper_globals)
            if "scrape" in scraper_globals:   # we have a scrape method
                scrape_method = scraper_globals["scrape"]
                callable(scrape_method) and scrape_method()    # call it

Answer 2

from sites import site_a,site_b,site_c
sites = [site_a,site_b,site_c]
for site in sites:
    site.scrape()

I guess might be what you are asking for

from sites import *
for item in globals():
    if item.startswith("site_") and hasattr(globals()[item],'scrape'):
       globals()[item].scrape()

introspection like this is kinda dicey though ... reader beware

Answer 3

您将需要使用inspect模块来处理此类事情。

import inspect
modules = [mod for mod in globals() if inspect.ismodule(eval(mod))]

将为您提供命名空间中作为模块的所有内容。如果需要，您应该能够看到如何对其进行更具体的修改。诀窍是运行eval将名称的字符串转换为对某个对象的引用，该对象可能是模块。

How to dynamically load python modules and call method on each of them?

3 个答案: