我在使用Apache Airflow时遇到问题。
启动时,它会记录许多连接,例如:
[2019-12-04 17:26:20,578] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,584] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,600] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,612] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,613] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,612] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,637] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,638] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,641] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,652] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,661] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,667] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,674] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,683] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,691] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,702] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,712] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,713] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,720] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,749] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,758] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,771] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,787] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,812] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,838] {{base_hook.py:83}} INFO - Using connection to: my_host
[2019-12-04 17:26:20,839] {{base_hook.py:83}} INFO - Using connection to: my_host
我的数据库连接超过50〜60,我只有12天。我已经检查了我的代码以查找是否没有关闭,但是我在关闭。
这是de Dags的照片,
这里有很多开放的Connections
我正在使用PostgresHook建立连接,这是我如何建立连接的示例代码:
# Recebendo variáveis de ambiente
src_con_id = os.getenv('A_ID')
database_src = os.getenv('A_DATABASE')
dest_con_id = os.getenv('B_ID')
database_redshift = os.getenv('B_DATABASE')
# SEMPRE COLOCAR CERTO O SCHEMA, O SRC É DA ONDE VEM A INFORMAÇÃO E O DEST É PRA ONDE VAI
table_schema_src = os.getenv('A_SCHEMA_DOMINIO')
table_schema_dest = os.getenv('B_SCHEMA_A_DOMINIO')
# Conexão src para a criação das tabelas
src_conn_teste = PostgresHook(postgres_conn_id=src_con_id, schema=database_src).get_conn()
dest_conn_teste = PostgresHook(postgres_conn_id=dest_con_id, schema=database_redshift).get_conn()
# Cursor para criação das tabelas
src_cursor_teste = src_conn_teste.cursor()
dest_cursor_teste = dest_conn_teste.cursor()