在Cloud TPU上训练对象检测器时，出现以下错误：
Error recorded from training_loop: Gradient for FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_5_1x1_48/weights:0 is NaN : Tensor had NaN values
这总是在我训练的同一步骤中发生。我完全不确定导致此错误的原因。
任何建议都会很棒！我会尽快答复。
我遵循this guide在Google的TPU系统上训练对象检测器。
这是完整的错误：
Error recorded from training_loop: Gradient for FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_5_1x1_48/weights:0 is NaN : Tensor had NaN values [[node CheckNumerics_99 (defined at /usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py:1112) ]] Caused by op u'CheckNumerics_99', defined at: File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/root/.local/lib/python2.7/site-packages/object_detection/model_tpu_main.py", line 142, in <module> tf.app.run() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "/root/.local/lib/python2.7/site-packages/object_detection/model_tpu_main.py", line 126, in main estimator.train(input_fn=train_input_fn, max_steps=train_steps) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2452, in train saving_listeners=saving_listeners) File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 358, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1154, in _train_model_default features, labels, model_fn_lib.ModeKeys.TRAIN, self.config) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2251, in _call_model_fn config) File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1112, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2633, in _model_fn update_ops = _sync_variables_ops(ctx) File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 207, in _sync_variables_ops for v in variables.trainable_variables() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 919, in check_numerics "CheckNumerics", tensor=tensor, message=message, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ self._traceback = tf_stack.extract_stack() InvalidArgumentError (see above for traceback): Gradient for FeatureExtractor/MobilenetV1/Conv2d_13_pointwise_1_Conv2d_5_1x1_48/weights:0 is NaN : Tensor had NaN values [[node CheckNumerics_99 (defined at /usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py:1112) ]]
Expand all | Collapse all {
答案 0 :(得分：0)
Tensorflow不能与NaN值配合使用。我建议使用内置的Imputer功能的SciKit Learns来“填写丢失的数据”
# Import Important Modules
import numpy as np # Numpy for Linear Algebra
import pandas as pd # Pandas for data storage and representation
import matplotlib.pyplot as plt # Matplotlib for glorious charts
from sklearn.impute import SimpleImputer
dataset = dataset
# This is a pretend variable. I don't know what the specifics of your
# dataset are
# Imputer has 2 "Strategies" you might find useful
# Mean (replacing the missing values in your dataset
# with the mean of all values in that column)
# Most Frequent (Replacing the missing values in your
# dataset with the most frequent value "I.E MODE" )
mean_imputer = SimpleImputer(missing_values=np.nan, strategy='mean')
# missing values parameter is what SimpleImputer is looking to
# "Fill" in the dataset
# strategy is the type of imputing SimpleImputer is going to perform.
# In this case, replace missing values with the mean of that column
most_frequent_imputer = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
### FIT THE IMPUTER TO YOUR DATASET:
# Note: One thing I didn't know for the longest time is that the
# methods I thought separate
# mean_imputer.fit() & mean_imputer.transform() can actually be
# performed in one action
# mean_imputer.fit_transform() :-)!
dataset = mean_imputer.fit_transform(dataset)
# Apply the mean imputer to the dataset
dataset = most_frequent_imputer.fit_transform(dataset)
# Apply the most frequent imputer to the dataset
### I hope this helps.
### If you have any questions don't hesitate to ask.