将气流日志写入Cloudwatch LogGroup

时间:2019-09-04 10:57:14

标签: terraform airflow amazon-cloudwatchlogs

我需要将所有Airflow dag日志写入CloudWatch Log组。

下面是我的var item = $(this).parent().attr('id'); 文件:-

awslogs.conf

[/var/log/dag_processor_manager] datetime_format = %b %d %H:%M:%S file = /home/ec2-user/airflow/logs/dag_processor_manager/dag_processor_manager.log buffer_duration = 5000 log_stream_name = {instance_id} initial_position = start_of_file log_group_name = airflow-log-group 以上,我可以看到.conf日志在CloudWatch LogGroup中写入,但是我需要dag日志,这些日志由于dag_processor_manager.log的配置而具有不同的文件夹结构。

airflow.cfg

logs.tf:-

# Log filename format
# we need to escape the curly braces by adding an additional curly brace
log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log

下面是文件夹结构:-

resource "aws_cloudwatch_log_group" "airflow_log_group" {
  name              = "airflow-log-group"
  retention_in_days = 30
}

如何更改awslogs.conf,使其可以添加此文件夹结构并获取所有日志。

0 个答案:

没有答案