为什么气流会持续不断地与气流数据库连接/断开连接?

时间:2019-04-23 18:32:38

标签: postgresql python-3.6 airflow

我看到许多气流实例连接到气流数据库,然后从气流数据库断开连接(在PostgreSQL 9.6上)。这些连接中的绝大多数甚至不会持续1/10秒,并且每秒最多可以有10个连接开始/结束。

此行为在几个不同的服务器上发生。两者都运行CentOS 7和PostgreSQL 9.6.12,气流为1.10.2,Anaconda python 3.6.5。 Postgres的最大连接数设置为200。

希望airflow.cfg最相关的部分:

executor = LocalExecutor
sql_alchemy_conn = postgresql://airflow@localhost:5432/airflow
sql_alchemy_pool_size = 32
sql_alchemy_pool_recycle = 3600
parallelism = 32
dag_concurrency = 32

PostgreSQL日志中的一个非常简短的示例:

2019-04-23 00:00:02 UTC [19856]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG:  connection received: host=::1 port=54460
2019-04-23 00:00:02 UTC [19856]: [2-1] user=airflow,db=airflow,app=[unknown],client=::1 LOG:  connection authorized: user=airflow database=airflow
2019-04-23 00:00:02 UTC [19855]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG:  connection received: host=::1 port=54458
2019-04-23 00:00:02 UTC [19855]: [2-1] user=airflow,db=airflow,app=[unknown],client=::1 LOG:  connection authorized: user=airflow database=airflow
2019-04-23 00:00:02 UTC [19857]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG:  connection received: host=::1 port=54462
2019-04-23 00:00:02 UTC [19857]: [2-1] user=airflow,db=airflow,app=[unknown],client=::1 LOG:  connection authorized: user=airflow database=airflow
2019-04-23 00:00:02 UTC [19858]: [1-1] user=[unknown],db=[unknown],app=[unknown],client=::1 LOG:  connection received: host=::1 port=54464
2019-04-23 00:00:02 UTC [19858]: [2-1] user=airflow,db=airflow,app=[unknown],client=::1 LOG:  connection authorized: user=airflow database=airflow
2019-04-23 00:00:02 UTC [19856]: [3-1] user=airflow,db=airflow,app=[unknown],client=::1 LOG:  disconnection: session time: 0:00:00.066 user=airflow database=airflow host=::1 port=54460
2019-04-23 00:00:02 UTC [19855]: [3-1] user=airflow,db=airflow,app=[unknown],client=::1 LOG:  disconnection: session time: 0:00:00.067 user=airflow database=airflow host=::1 port=54458
2019-04-23 00:00:02 UTC [19857]: [3-1] user=airflow,db=airflow,app=[unknown],client=::1 LOG:  disconnection: session time: 0:00:00.091 user=airflow database=airflow host=::1 port=54462

我只是想知道可以做些什么来限制这项活动,或者至少我想了解这项活动的目的是什么。

非常感谢您的帮助。

0 个答案:

没有答案