Question

由于某种原因，python logger格式有时会损坏。我不确定是什么问题，看起来像是编码问题：

I0716 23:27:10.491452 4409853376 tpu_context.py:209] _TPUContext: eval_on_tpu True
W0716 23:27:10.491577 4409853376 tpu_context.py:211] eval_on_tpu ignored because use_tpu is False.
W0716 23:27:10.619174 4409853376 deprecation_wrapper.py:119] From bert-ner.py:423: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

W0716 23:27:10.621710 4409853376 deprecation_wrapper.py:119] From bert-ner.py:428: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

应如下所示：

INFO ...
INFO ...
WARN ...
WARN ...

脚本的执行方式如下：

subprocess.call('python3 bert-ner.py ...', shell=True)

该如何解决？

Answer 1

Python API

如果您只想自定义tensorflow的日志记录格式，请在absl和tensorflow记录器中替换格式化程序：

import logging
from absl import logging as absl_logging
import tensorflow as tf


fmt = '[%(levelname)s %(asctime)s %(filename)s:%(lineno)s] %(message)s'
formatter = logging.Formatter(fmt)

absl_logging.get_absl_handler().setFormatter(formatter)
absl_logging.set_verbosity('debug')  # for example

for h in tf.get_logger().handlers:
    h.setFormatter(formatter)
tf.compat.v1.logging.set_verbosity(logging.DEBUG)  # for example


# test configuration
if __name__ == '__main__':
    # test custom logger if you have one - should also emit in the same format
    logging.getLogger(__name__).debug('tf imported')
    # this issues a DeprecationWarning in tensorflow>=1.13, emitting a warning log message
    s = tf.Session()

脚本发出：

WARNING: Logging before flag parsing goes to stderr.
[DEBUG 2019-07-18 14:03:15,662 eggs.py:20] tf imported
[WARNING 2019-07-18 14:03:15,662 deprecation_wrapper.py:119] From eggs.py:22: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2019-07-18 14:03:15.671392: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-18 14:03:15.674517: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 1992000000 Hz
...

C ++ API

Python部件发出的日志记录现在具有不同的统一输出。但是，如您在上面的输出中看到的，从C ++部件发出的日志不受此影响。不幸的是，截至目前（2019年7月18日），C ++代码中的日志格式不可配置，请参见LogMessage::GenerateLogMessage() impl：

void LogMessage::GenerateLogMessage() {
  ...
  strftime(time_buffer, time_buffer_size, "%Y-%m-%d %H:%M:%S",
           localtime(&now_seconds));

  // TODO(jeff,sanjay): Replace this with something that logs through the env.
  fprintf(stderr, "%s.%06d: %c %s:%d] %s\n", time_buffer, micros_remainder,
          "IWEF"[severity_], fname_, line_, str().c_str());
}

这意味着除了关闭库的C ++部分的日志发射之外，您无法做其他事情：

import logging
import os
from absl import logging as absl_logging
import tensorflow as tf

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

...

记录对齐方式

如果您像我一样，又不希望日志行未正确对齐，请对fmt字符串进行一些修改：

fmt = '[%(levelname)8s %(asctime)s %(filename)s:%(lineno)s] %(message)s'

现在输出将更具可读性：

[   DEBUG 2019-07-18 14:31:20,097 eggs.py:36] tf imported
[ WARNING 2019-07-18 14:31:20,097 deprecation_wrapper.py:119] From eggs.py:38: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

使用微秒和线程ID

默认的abseil处理程序会在日期之后加上微秒，并且还包括当前线程的ID。您也可以这样做-线程ID由logging模块提供，而必须首先计算微秒（为此我使用了自定义Filter）。再来看上面的示例脚本：

import logging
import os
import time
from absl import logging as absl_logging
import tensorflow as tf

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

fmt = '[%(levelname)8s %(asctime)s.%(microsecs)06d %(thread)d %(filename)s:%(lineno)s] %(message)s'
date_fmt = '%Y-%m-%d %H:%M:%S'
formatter = logging.Formatter(fmt, date_fmt)


class FilterMicroseconds(logging.Filter):
    def filter(self, record):
        local_time = time.localtime(record.created)
        record.microsecs = int(record.created % 1.0 * 1e6)
        return True

filter_microsecs = FilterMicroseconds()


absl_logging.get_absl_handler().setFormatter(formatter)
absl_logging.get_absl_handler().addFilter(filter_microsecs)
absl_logging.set_verbosity('debug')  # for example

for h in tf.get_logger().handlers:
    h.setFormatter(formatter)
    h.addFilter(filter_microsecs)
tf.compat.v1.logging.set_verbosity(logging.DEBUG)  # for example


# test configuration
if __name__ == '__main__':
    # test custom logger if you have one - should also emit in the same format
    logging.getLogger(__name__).debug('tf imported')
    # this issues a DeprecationWarning in tensorflow>=1.13, emitting a warning log message
    s = tf.Session()

输出：

WARNING: Logging before flag parsing goes to stderr.
[   DEBUG 2019-07-18 14:39:04.522035 140546459115328 eggs.py:36] tf imported
[ WARNING 2019-07-18 14:39:04.522186 140546459115328 deprecation_wrapper.py:119] From eggs.py:38: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

Python记录器格式损坏：I0716而不是INFO

1 个答案:

Python API

C ++ API

记录对齐方式

使用微秒和线程ID