Question

我无法使用systemd来启动气流Web服务器，即使它已启动并在systemd之外正常运行，如下所示：

export AIRFLOW_HOME=/path/to/my/airflow/home ; airflow webserver -p 8080

systemd日志使我相信问题出在gunicorn，即使当我运行上述命令时gunicorn开始运行时也没有问题（即，这只是systemd中的问题）。我已经根据气流文档（运行Ubuntu 16）配置了以下systemd文件。

/ etc / default / airflow

AIRFLOW_HOME=/path/to/my/airflow/home
SCHEDULER_RUNS=5

/lib/systemd/system/airflow-webserver.service

[Unit]
Description=Airflow webserver daemon   
After=network.target

[Service]
EnvironmentFile=/etc/default/airflow
User=ubuntu
Group=ubuntu
Type=simple
ExecStart=/bin/bash -c "export AIRFLOW_HOME=/path/to/my/airflow/home ; airflow webserver -p 8080 "

Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

/etc/tmpfiles.d/airflow.conf

D /run/airflow 0755 airflow airflow

当我使用systemctl启动服务时，这导致以下错误。

systemctl start airflow-webserver.service

Jul 15 22:41:27 ip-172-31-19-64 systemd[1]: Started Airflow webserver daemon.
Jul 15 22:41:27 ip-172-31-19-64 bash[31494]: [2018-07-15 22:41:27,555] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt
Jul 15 22:41:27 ip-172-31-19-64 bash[31494]: [2018-07-15 22:41:27,592] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt
Jul 15 22:41:27 ip-172-31-19-64 bash[31494]: [2018-07-15 22:41:27,729] {__init__.py:45} INFO - Using executor SequentialExecutor
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:   ____________       _____________
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:  ____    |__( )_________  __/__  /________      __
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:  _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: [2018-07-15 22:41:28,042] {models.py:189} INFO - Filling up the DagBag from /path/to/my/airflow/home/dags
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: /home/ubuntu/.local/lib/python3.5/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.cache is deprecated, use flask_cach
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:   .format(x=modname), ExtDeprecationWarning
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: Running the Gunicorn Server with:
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: Workers: 4 sync
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: Host: 0.0.0.0:8080
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: Timeout: 120
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: Logfiles: - -
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: =================================================================
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: Traceback (most recent call last):
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:   File "/usr/local/bin/airflow", line 27, in <module>
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:     args.func(args)
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 788, in webserver
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:     gunicorn_master_proc = subprocess.Popen(run_args)
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:   File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:     restore_signals, start_new_session)
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:   File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]:     raise child_exception_type(errno_num, err_msg)
Jul 15 22:41:28 ip-172-31-19-64 bash[31494]: FileNotFoundError: [Errno 2] No such file or directory: 'gunicorn'
Jul 15 22:41:28 ip-172-31-19-64 systemd[1]: airflow-webserver.service: Main process exited, code=exited, status=1/FAILURE

我需要做一些配置来使gunicorn与systemd兼容吗？

编辑：遵循有关这是权限问题的建议，我通过以下方式安装了gunicorn：sudo apt-get install gunicorn，并且在重新运行systemctl时收到以下错误Error: No module named airflow.www.gunicorn_config。我认为这是由于我刚安装的gunicorn和我的ubuntu用户用来运行气流的gunicorn之间不一致，所以我用前者替换了/ usr / bin /中的gunicorn。此修补程序可能不是进行此修补的最佳方法，但此后我就能够通过systemd成功运行气流。

Answer 1

我在/ srv / airflow下的虚拟环境中安装的Ubuntu 18.04 LTS和Apache Airflow版本1.10.1上遇到了相同的问题。经过大量的试验和错误，我最终得到了这个可行的解决方案。

我的airflow-webserver.service文件：

[Unit]
Description=Airflow webserver daemon
After=network.target

[Service]
Environment="PATH=/srv/airflow/bin"
Environment="AIRFLOW_HOME=/srv/airflow"
User=airflow
Group=airflow
Type=simple
ExecStart=/srv/airflow/bin/airflow webserver --pid /srv/airflow/webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

我这样做是为了安装服务：

sudo cp airflow-webserver.service /lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable airflow-webserver.service
sudo systemctl start airflow-webserver.service

Answer 2

是的，ExecStart=..airflow/bin/airflow 需要 ..airflow/bin/python 但 systemd 找到了 system python。气流始终需要 Environment="PATH=..airflow/bin"。

Answer 3

在Ubuntu Bionic中，我发现在root python环境中先使用sudo apt-get install python3-gunicorn然后再使用sudo apt-get install gunicorn可以解决此问题。

由于喷枪而导致气流系统失败

3 个答案: