气流Docker映像未按预期处理Python模块

时间:2019-06-28 17:56:23

标签: python docker airflow

我正在使用puckel/docker-airflow和经过稍微修改的docker-compose在AWS中运行docker-compose.yml,并且没有按照我的期望选择Python模块,我想知道是否可以我做错了什么,或者我的说法不正确。当我尝试在DAG文件中使用Python模块(例如smart_open)时,出现错误消息:

webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/airflow/models/__init__.py", line 413, in process_file
webserver_1  |     m = imp.load_source(mod_name, filepath)
webserver_1  |   File "/usr/local/lib/python3.6/imp.py", line 172, in load_source
webserver_1  |     module = _load(spec)
webserver_1  |   File "<frozen importlib._bootstrap>", line 684, in _load
webserver_1  |   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/airflow/dags/mysql_etl.py", line 11, in <module>
webserver_1  |     import smart_open
webserver_1  | ModuleNotFoundError: No module named 'smart_open'

但是,我在smart_open中列出了requirements.txt,在我的requirements.txt中已经装载了docker-compose.yml文件,并且该文件似乎已按预期加载:

webserver_1  | Collecting smart_open (from -r /requirements.txt (line 3))
webserver_1  |   Downloading https://files.pythonhosted.org/packages/37/c0/25d19badc495428dec6a4bf7782de617ee0246a9211af75b302a2681dea7/smart_open-1.8.4.tar.gz (63kB)

或者,当我使用以下命令构建smart_open的docker镜像时,错误消失了 --build-arg PYTHON_DEPS="smart_open"

就我所见,默认行为执行了entrypoint.sh并按预期运行,因此我没有从puckel/docker-airflow修改pip install --user -r requirements.txt

因此:我需要对DAG文件进行其他操作以访问requirements.txt中指定的Python模块吗?还是仅使用所有所需模块构建Docker映像?感谢您对此的任何见识!

0 个答案:

没有答案