Question

我正在尝试在AWS Sagemaker上训练TensorFlow模型。我为此创建了带有外部库的容器（在Amazon SageMaker中使用您自己的算法或模型）。

我们使用TensorFlow API进行培训

from sagemaker.tensorflow import TensorFlow
estimator = TensorFlow(
  entry_point="entry.py",             # entry script
  role=role,
  framework_version="1.13.0",   
  py_version='py3',
  hyperparameters=hyperparameters,
  train_instance_count=1,                   # "The number of GPUs instances to use"
  train_instance_type=train_instance_type,
  image_name=my_image

)
estimator.fit({'train': train_s3, 'eval': eval_s3})

并出现错误：

09:06:46
2019-07-23 09:06:45,463 INFO - root - running container entrypoint 
09:06:46
2019-07-23 09:06:45,463 INFO - root - starting train task 
09:06:46
2019-07-23 09:06:45,476 INFO - container_support.training - Training starting 
09:06:46
2019-07-23 09:06:45,479 ERROR - container_support.training - uncaught exception during training: No module named 'tf_container'

09:06:46
Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/container_support/environment.py", line 136, in load_framework return importlib.import_module('mxnet_container') File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 994, in _gcd_i 
09:06:46
ModuleNotFoundError: No module named 'mxnet_container'

09:06:46
During handling of the above exception, another exception occurred:

09:06:46
Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/container_support/training.py", line 35, in start fw = TrainingEnvironment.load_framework() File "/usr/local/lib/python3.6/dist-packages/container_support/environment.py", line 138, in load_framework return importlib.import_module('tf_container') File "/usr/lib/python3.6/importlib/__init__.py", line 126, 
09:06:46
ModuleNotFoundError: No module named 'tf_container'

我该怎么做才能解决此问题？我该如何调试这种情况？

Answer 1

我猜测您使用的是自己的TF容器，而不是https://github.com/aws/sagemaker-tensorflow-container的SageMaker容器

如果是这种情况，则您的容器缺少使用TensorFlow估算器（“ tf_container”软件包）所需的支持代码。

解决方案是从SageMaker容器开始，对其进行自定义，将其推回ECR，然后将图像名称通过'image_name'参数传递给SageMaker估算器。

在Sagemaker上使用TensorFlow进行培训没有名为'tf_container'的模块

1 个答案: