Question

我在Python 3.5上使用了airflow 1.10.0，并遇到了有关日志记录编码错误的错误。

操作员使用output_encoding的默认设置，该设置已经是utf-8。

task_compile = BashOperator(
task_id='task_compile',
retries=1,
retry_delay=timedelta(minutes=5),
bash_command='/root/docker/tools/compile.sh',
dag=dag
)
task_compile.set_downstream(task_last)

shell脚本会弹出一个docker容器并运行composer安装，我用另一个简单的composer安装任务进行了测试，但没有任何失败，该错误仅与某些依赖项有关。如跟踪堆栈中所示，负责异常的模块是 file_task_handler.py ，它发出要记录到日志文件中的行。

[2018-09-19 20:42:18,708] {bash_operator.py:111} INFO - Package operations: 134 installs, 0 updates, 0 removals
[2018-09-19 20:42:18,790] {bash_operator.py:111} INFO -   - Installing ocramius/package-versions (1.3.0): Downloading (100%)
[2018-09-19 20:42:18,850] {bash_operator.py:111} INFO -   - Installing symfony/flex (v1.1.1): Downloading (100%)
[2018-09-19 20:42:18,897] {bash_operator.py:111} INFO - 
[2018-09-19 20:42:18,898] {logging_mixin.py:95} WARNING - --- Logging error ---

[2018-09-19 16:12:51,554] {logging_mixin.py:95} WARNING - --- Logging error ---
[2018-09-19 16:12:51,555] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2018-09-19 16:12:51,555] {logging_mixin.py:95} WARNING -   File "/usr/lib/python3.5/logging/__init__.py", line 983, in emit
    stream.write(msg)
[2018-09-19 16:12:51,555] {logging_mixin.py:95} WARNING - UnicodeEncodeError: 'ascii' codec can't encode character '\U0001f3b6' in position 81: ordinal not in range(128)
[2018-09-19 16:12:51,555] {logging_mixin.py:95} WARNING - Call stack:
[2018-09-19 16:12:51,557] {logging_mixin.py:95} WARNING -   File "/usr/local/bin/airflow", line 32, in <module>
    args.func(args)
[2018-09-19 16:12:51,557] {logging_mixin.py:95} WARNING -   File "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
[2018-09-19 16:12:51,557] {logging_mixin.py:95} WARNING -   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 498, in run
    _run(args, dag, ti)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 402, in _run
    pool=args.pool,
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/local/lib/python3.5/dist-packages/airflow/utils/db.py", line 74, in wrapper
    return func(*args, **kwargs)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1633, in _run_raw_task
    result = task_copy.execute(context=context)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/local/lib/python3.5/dist-packages/airflow/operators/bash_operator.py", line 110, in execute
    self.log.info(line)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/lib/python3.5/logging/__init__.py", line 1280, in info
    self._log(INFO, msg, args, **kwargs)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/lib/python3.5/logging/__init__.py", line 1416, in _log
    self.handle(record)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/lib/python3.5/logging/__init__.py", line 1426, in handle
    self.callHandlers(record)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/lib/python3.5/logging/__init__.py", line 1488, in callHandlers
    hdlr.handle(record)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/lib/python3.5/logging/__init__.py", line 856, in handle
    self.emit(record)
[2018-09-19 16:12:51,558] {logging_mixin.py:95} WARNING -   File "/usr/local/lib/python3.5/dist-packages/airflow/utils/log/file_task_handler.py", line 61, in emit
    self.handler.emit(record)

Answer 1

问题在于容器中的语言环境未设置为UTF-8

Answer 2

面对类似的问题，我可以通过将环境变量LANG=en_US.UTF-8添加到受监控的配置中并重新启动受监控的来解决此问题。

我使用主管启动气流调度程序，Web服务器和鲜花。

注意：此env变量也需要添加到所有气流工作节点中。

气流BashOperator UnicodeEncodeError

2 个答案: