我们有10个气流DAG,我们使用条件触发器运行下一个DAG。但是下一个DAG并没有立即开始。触发正好需要5分钟。
示例:
Dag1于10:05完成 Dag2于10:10开始
如果我们能够消除5分钟的差距,整个ETL过程将在2.5小时内完成。但现在需要大约3到4个小时。
配置文件:
sql_alchemy_pool_size = 5
sql_alchemy_pool_recycle = 3600
parallelism = 8
dag_concurrency = 64
dags_are_paused_at_creation = True
max_active_runs_per_dag = 16
load_examples = False
donot_pickle = False
dagbag_import_timeout = 30
workers = 4
worker_class = sync
authenticate = False
filter_by_owner = False
email_backend = airflow.utils.send_email_smtp
celery_app_name = airflow.executors.celery_executor
celeryd_concurrency = 16
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5
task_cpu = 1