当我运行 RLlib 训练器时，您将在下面看到日志输出中的前几行相关行，希望看到生成的 TfEvent 文件，以便我可以在 TensorBoard 上查看它们。
我使用的基本 docker 映像是 tensorflow/tensorflow:latest-gpu-py3，它给我 TensorFlow 2.5，我使用 Ray 1.3.0 进行训练。
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/compat/v2_compat.py:88: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
/usr/local/lib/python3.6/dist-packages/ray/autoscaler/_private/cli_logger.py:61: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via `pip install 'ray[default]'`. Please update your install command.
"update your install command.", FutureWarning)
2021-06-27 20:44:37,007 INFO services.py:1269 -- View the Ray dashboard at http://127.0.0.1:8265
/usr/local/lib/python3.6/dist-packages/gym/logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
2021-06-27 20:44:38,828 INFO logger.py:180 -- pip install 'ray[tune]' to see TensorBoard files.
2021-06-27 20:44:38,828 WARNING logger.py:317 -- Could not instantiate TBXLogger: No module named 'torch'.
2021-06-27 20:44:38,834 INFO trainer.py:669 -- Tip: set framework=tfe or the --eager flag to enable TensorFlow eager execution
(pid=211) WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/compat/v2_compat.py:88: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
(pid=211) Instructions for updating:
(pid=211) non-resource variables are not supported in the long term
(pid=211) 2021-06-27 20:44:40,339 INFO rollout_worker.py:497 -- Could not seed torch
(pid=211) 2021-06-27 20:44:40,390 INFO catalog.py:387 -- Wrapping <class 'my_vision_net.MyVisionNetwork'> as None
(pid=211) WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
(pid=211) Instructions for updating:
(pid=211) If using Keras pass *_constraint arguments to layers.
(pid=211) WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
(pid=211) Instructions for updating:
(pid=211) If using Keras pass *_constraint arguments to layers.
(pid=211) Constructing vision network...
(pid=211) 2021-06-27 20:44:40,612 INFO dynamic_tf_policy.py:463 -- Testing `compute_actions` w/ dummy batch.
(pid=211) 2021-06-27 20:44:40,612 INFO tf_run_builder.py:87 -- Executing TF run without tracing. To dump TF timeline traces to disk, set the TF_TIMELINE_DIR environment variable.
(pid=211) 2021-06-27 20:44:40,678 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_prob` to view-reqs.
(pid=211) 2021-06-27 20:44:40,678 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_logp` to view-reqs.
(pid=211) 2021-06-27 20:44:40,679 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_dist_inputs` to view-reqs.
(pid=211) 2021-06-27 20:44:40,679 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `vf_preds` to view-reqs.
(pid=211) 2021-06-27 20:44:40,679 INFO dynamic_tf_policy.py:479 -- Testing `postprocess_trajectory` w/ dummy batch.
2021-06-27 20:44:41,162 INFO rollout_worker.py:490 -- Env doesn't support env.seed(): None
2021-06-27 20:44:41,162 INFO rollout_worker.py:497 -- Could not seed torch
2021-06-27 20:44:41,450 INFO catalog.py:387 -- Wrapping <class 'my_vision_net.MyVisionNetwork'> as None
Constructing vision network...
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
2021-06-27 20:44:42,235 INFO dynamic_tf_policy.py:463 -- Testing `compute_actions` w/ dummy batch.
2021-06-27 20:44:42,235 INFO tf_run_builder.py:87 -- Executing TF run without tracing. To dump TF timeline traces to disk, set the TF_TIMELINE_DIR environment variable.
2021-06-27 20:44:43,892 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_prob` to view-reqs.
2021-06-27 20:44:43,893 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_logp` to view-reqs.
2021-06-27 20:44:43,893 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_dist_inputs` to view-reqs.
2021-06-27 20:44:43,894 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `vf_preds` to view-reqs.
2021-06-27 20:44:43,894 INFO dynamic_tf_policy.py:479 -- Testing `postprocess_trajectory` w/ dummy batch.
2021-06-27 20:44:44,455 INFO rollout_worker.py:1161 -- Built policy map: {'my_ppo': <ray.rllib.policy.tf_policy_template.PPOTFPolicy object at 0x7f5da47b3438>}
2021-06-27 20:44:44,455 INFO rollout_worker.py:1162 -- Built preprocessor map: {'my_ppo': <ray.rllib.models.preprocessors.NoPreprocessor object at 0x7f5da47b30b8>}
2021-06-27 20:44:44,456 INFO rollout_worker.py:563 -- Built filter map: {'my_ppo': <ray.rllib.utils.filter.NoFilter object at 0x7f5da479af60>}
2021-06-27 20:44:44,475 WARNING deprecation.py:34 -- DeprecationWarning: `simple_optimizer` has been deprecated. This will raise an error in the future!
(pid=202) WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/compat/v2_compat.py:88: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
(pid=202) Instructions for updating:
(pid=202) non-resource variables are not supported in the long term
(pid=202) 2021-06-27 20:44:45,909 INFO rollout_worker.py:497 -- Could not seed torch
(pid=202) 2021-06-27 20:44:45,941 INFO catalog.py:387 -- Wrapping <class 'my_vision_net.MyVisionNetwork'> as None
(pid=202) Constructing vision network...
(pid=202) WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
(pid=202) Instructions for updating:
(pid=202) If using Keras pass *_constraint arguments to layers.
(pid=202) WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1635: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
(pid=202) Instructions for updating:
(pid=202) If using Keras pass *_constraint arguments to layers.
(pid=202) 2021-06-27 20:44:46,118 INFO dynamic_tf_policy.py:463 -- Testing `compute_actions` w/ dummy batch.
(pid=202) 2021-06-27 20:44:46,118 INFO tf_run_builder.py:87 -- Executing TF run without tracing. To dump TF timeline traces to disk, set the TF_TIMELINE_DIR environment variable.
(pid=202) 2021-06-27 20:44:46,164 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_prob` to view-reqs.
(pid=202) 2021-06-27 20:44:46,165 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_logp` to view-reqs.
(pid=202) 2021-06-27 20:44:46,165 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `action_dist_inputs` to view-reqs.
(pid=202) 2021-06-27 20:44:46,165 INFO dynamic_tf_policy.py:472 -- Adding extra-action-fetch `vf_preds` to view-reqs.
(pid=202) 2021-06-27 20:44:46,166 INFO dynamic_tf_policy.py:479 -- Testing `postprocess_trajectory` w/ dummy batch.
2021-06-27 20:44:46,657 INFO rollout_worker.py:490 -- Env doesn't support env.seed(): None
2021-06-27 20:44:46,657 INFO rollout_worker.py:497 -- Could not seed torch
2021-06-27 20:44:46,661 INFO catalog.py:387 -- Wrapping <class 'my_vision_net.MyVisionNetwork'> as None
我可以看到主要问题是线路：
2021-06-27 20:44:38,828 WARNING logger.py:317 -- Could not instantiate TBXLogger: No module named 'torch'.
当我使用 TensorFlow 时，我不明白为什么它甚至需要 Torch。另外，我真的需要它来生成那些 TFevent 文件。这里有一个类似的问题 (https://github.com/ray-project/ray/issues/5546)，但我不能在这种情况下应用相同的解决方案。
编辑（相关代码供参考）：
def custom_log_creator(custom_path, custom_str):
timestr = datetime.today().strftime("%Y-%m-%d_%H-%M-%S")
logdir_prefix = "{}_{}".format(custom_str, timestr)
def logger_creator(config):
if not os.path.exists(custom_path):
os.makedirs(custom_path) os.makedirs(os.path.join(custom_path,logdir_prefix),exist_ok=True)
logdir = os.path.join(custom_path,logdir_prefix)
return UnifiedLogger(config, logdir, loggers=None)
return logger_creator
log_path = os.path.join(pathlib.Path(__file__).parent.absolute(), 'results')
trainer = ppo.PPOTrainer(env='test_env',
config=policy_conf,
logger_creator=custom_log_creator(custom_path=log_path, custom_str='res'))