Airflow Worker SQL连接丢失

时间:2018-11-29 15:50:18

标签: python-3.x sqlalchemy airflow ubuntu-18.04

我是Apache-Airflow的新手,在正确设置它时遇到了问题。我正在Windows 10计算机上的Linux子系统中使用Python 3.6.6。

leolei@PC-W10E277:~$ cat /etc/issue
Ubuntu 18.04.1 LTS \n \l
leolei@PC-W10E277:~$ cat /proc/version
Linux version 4.4.0-17134-Microsoft (Microsoft@Microsoft.com) (gcc version 5.4.0 (GCC) ) #345-Microsoft Wed Sep 19 17:47:00 PST 2018

Web服务器和调度程序正在运行,没有错误,也没有配置。

我安装了研究期间发现的所有内容(请参阅底部的点冻结和软件包列表)。

我相信它很容易修复,但是没有其他人有这个确切的问题。

当我尝试通过“气流工作程序”启动工作程序而不进行配置时,会发生这种情况: 它说它失去了连接,然后在几秒钟后再次尝试并失去了连接,依此类推。

在此感谢我如何工作的所有提示,我已经在此浪费了两个工作日。

跟踪:

leolei@PC-W10E277:~$ airflow worker
[2018-11-29 16:14:09,385] {__init__.py:51} INFO - Using executor SequentialExecutor

 -------------- celery@PC-W10E277 v4.1.1 (latentcall)
---- **** -----
--- * ***  * -- Linux-4.4.0-17134-Microsoft-x86_64-with-Ubuntu-18.04-bionic 2018-11-29 16:14:09
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         airflow.executors.celery_executor:0x7f5856d0d438
- ** ---------- .> transport:   sqla+mysql://airflow:airflow@localhost:3306/airflow
- ** ---------- .> results:     mysql://airflow:**@localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> default          exchange=default(direct) key=default


[2018-11-29 16:14:10,207: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/home/leolei/.local/lib/python3.6/site-packages/sqlalchemy/pool.py", line 1124, in _do_get
    return self._pool.get(wait, self._timeout)
  File "/home/leolei/.local/lib/python3.6/site-packages/sqlalchemy/util/queue.py", line 145, in get
    raise Empty
sqlalchemy.util.queue.Empty

During handling of the above exception, another exception occurred:
[...]

点冻结:

leolei@PC-W10E277:~$ pip freeze
adal==1.2.0
alembic==0.8.10
amqp==2.3.2
apache-airflow==1.10.0
asn1crypto==0.24.0
attrs==17.4.0
Automat==0.6.0
Babel==2.6.0
billiard==3.5.0.4
bleach==2.1.2
blinker==1.4
cachetools==3.0.0
celery==4.1.1
certifi==2018.1.18
chardet==3.0.4
click==6.7
cloud-init==18.4
colorama==0.4.0
command-not-found==0.3
configobj==5.0.6
configparser==3.5.0
constantly==15.1.0
croniter==0.3.26
cryptography==2.1.4
defusedxml==0.5.0
dill==0.2.8.2
distro-info==0.18
docutils==0.14
Flask==0.12.4
Flask-Admin==1.4.1
Flask-AppBuilder==1.12.2
Flask-Babel==0.12.2
Flask-Caching==1.3.3
Flask-Login==0.2.11
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.3.2
flask-swagger==0.2.13
Flask-WTF==0.14.2
flower==0.9.2
funcsigs==1.0.0
future==0.16.0
gevent==1.3.7
gitdb2==2.0.5
GitPython==2.1.11
google-auth==1.6.1
greenlet==0.4.15
gunicorn==19.9.0
html5lib==1.0.1
httplib2==0.9.2
hyperlink==17.3.1
idna==2.6
incremental==16.10.1
iso8601==0.1.12
itsdangerous==1.1.0
Jinja2==2.8.1
jsonpatch==1.16
jsonpointer==1.10
jsonschema==2.6.0
kombu==4.2.1
kubernetes==8.0.0
language-selector==0.1
librabbitmq==2.0.0
lockfile==0.12.2
lxml==3.8.0
Mako==1.0.7
Markdown==2.6.11
MarkupSafe==1.0
monotonic==1.5
mysql-connector-python==8.0.13
mysqlclient==1.3.10
numpy==1.15.4
oauthlib==2.0.6
ordereddict==1.1
PAM==0.4.2
pandas==0.23.4
pendulum==1.4.4
protobuf==3.6.1
psutil==4.4.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
Pygments==2.2.0
pygobject==3.26.1
PyJWT==1.5.3
pyOpenSSL==17.5.0
pyserial==3.4
python-apt==1.6.3
python-daemon==2.1.2
python-dateutil==2.7.5
python-debian==0.1.32
python-editor==1.0.3
python-nvd3==0.15.0
python-slugify==1.2.6
python3-openid==3.1.0
pytz==2018.7
pytzdata==2018.7
PyYAML==3.12
redis==3.0.1
requests==2.20.1
requests-oauthlib==1.0.0
requests-unixsocket==0.1.5
rsa==4.0
service-identity==16.0.0
setproctitle==1.1.10
six==1.11.0
smmap2==2.0.5
SQLAlchemy==1.1.18
ssh-import-id==5.7
systemd-python==234
tabulate==0.7.7
tenacity==4.8.0
thrift==0.11.0
tornado==5.1.1
Twisted==17.9.0
tzlocal==1.5.1
ufw==0.35
unattended-upgrades==0.1
unicodecsv==0.14.1
Unidecode==1.0.22
urllib3==1.22
vine==1.1.4
webencodings==0.5.1
websocket-client==0.54.0
Werkzeug==0.14.1
WTForms==2.2.1
zope.deprecation==4.3.0
zope.interface==4.3.2

已安装的sql软件包:

leolei@PC-W10E277:~$ apt list --installed *sql*
Listing... Done
libmysqlclient20/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
libsqlite3-0/bionic,now 3.22.0-1 amd64 [installed]
mysql-client/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 all [installed]
mysql-client-5.7/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
mysql-client-core-5.7/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
mysql-common/bionic,now 5.8+1.0.4 all [installed]
mysql-server/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 all [installed]
mysql-server-5.7/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
mysql-server-core-5.7/bionic-updates,bionic-security,now 5.7.24-0ubuntu0.18.04.1 amd64 [installed,automatic]
python3-mysqldb/bionic,now 1.3.10-1build1 amd64 [installed]
python3-sqlalchemy/bionic,now 1.1.11+ds1-1ubuntu1 all [installed]

1 个答案:

答案 0 :(得分:1)

我认为答案是:Windows Linux子系统无法实现。我不知道为什么。