我正在使用puckel/docker-airflow
和经过稍微修改的docker-compose
在AWS中运行docker-compose.yml
,并且没有按照我的期望选择Python模块,我想知道是否可以我做错了什么,或者我的说法不正确。当我尝试在DAG文件中使用Python模块(例如smart_open
)时,出现错误消息:
webserver_1 | Traceback (most recent call last):
webserver_1 | File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 413, in process_file
webserver_1 | m = imp.load_source(mod_name, filepath)
webserver_1 | File "/usr/local/lib/python3.6/imp.py", line 172, in load_source
webserver_1 | module = _load(spec)
webserver_1 | File "<frozen importlib._bootstrap>", line 684, in _load
webserver_1 | File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
webserver_1 | File "<frozen importlib._bootstrap_external>", line 678, in exec_module
webserver_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1 | File "/usr/local/airflow/dags/mysql_etl.py", line 11, in <module>
webserver_1 | import smart_open
webserver_1 | ModuleNotFoundError: No module named 'smart_open'
但是,我在smart_open
中列出了requirements.txt
,在我的requirements.txt
中已经装载了docker-compose.yml
文件,并且该文件似乎已按预期加载:>
webserver_1 | Collecting smart_open (from -r /requirements.txt (line 3))
webserver_1 | Downloading https://files.pythonhosted.org/packages/37/c0/25d19badc495428dec6a4bf7782de617ee0246a9211af75b302a2681dea7/smart_open-1.8.4.tar.gz (63kB)
或者,当我使用以下命令构建smart_open
的docker镜像时,错误消失了
--build-arg PYTHON_DEPS="smart_open"
就我所见,默认行为执行了entrypoint.sh
并按预期运行,因此我没有从puckel/docker-airflow
修改pip install --user -r requirements.txt
。
因此:我需要对DAG文件进行其他操作以访问requirements.txt
中指定的Python模块吗?还是仅使用所有所需模块构建Docker映像?感谢您对此的任何见识!