TensorFlow v2:CancelledError:[Op:StatefulPartitionedCall]

时间:2019-06-21 23:12:13

标签: python tensorflow machine-learning keras

在玩TensorFlow v2模型并尝试对其进行训练时,我不断出现此错误:

  

CancelledError:[Op:StatefulPartitionedCall]

我正在jupyter笔记本上运行此程序,并在CPU上使用了最新的beta版本的tensorflow v2。

我的代码是:

def training_loop(model, data, num_epochs=10):

    def loss(x, y):
        error = tf.square(tf.subtract(model.predict(x), y))
        return error

    def grad(x, y):
        with tf.GradientTape() as tape:
            loss_value = loss(x, y)
        return loss_value, tape.gradient(loss_value, model.trainable_variables)

    learning_rate = 0.002
    optimizer = tf.train.AdamOptimizer(learning_rate)


    train_loss_results = []

    for epoch in range(num_epochs):

        epoch_loss_avg = tf.keras.metrics.Mean()

        # Training loop - using batches of 64
        for x, y in data:

            # Optimize the model
            loss_value, grads = grad(x.numpy(), y.numpy())
            optimizer.apply_gradients(zip(grads, model.trainable_variables))

        # Track progress
        epoch_loss_avg(loss_value)  # add current batch loss
        # compare predicted label to actual label

        # end epoch
        train_loss_results.append(epoch_loss_avg.result())

        if epoch % 50 == 0:
            print('Epoch {:03d}: Loss: {:.3f}'.format(epoch, epoch_loss_avg.result()))

当我尝试运行training_loop时,出现上述错误。

完整的回溯是:

CancelledError Traceback (most recent call last)
<ipython-input-49-57231385902d> in <module>
----> 1 p = ae.predict(x1)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in predict(self, x, batch_size, verbose, steps, max_queue_size, workers, use_multiprocessing)
1111 else:
1112 return training_arrays.predict_loop(
-> 1113 self, x, batch_size=batch_size, verbose=verbose, steps=steps)
1114
1115 def reset_metrics(self):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training_arrays.py in model_iteration(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps, mode, validation_in_fit, **kwargs)
327
328 # Get outputs.
--> 329 batch_outs = f(ins_batch)
330 if not isinstance(batch_outs, list):
331 batch_outs = [batch_outs]

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py in __call__(self, inputs)
3164 value = math_ops.cast(value, tensor.dtype)
3165 converted_inputs.append(value)
-> 3166 outputs = self._graph_fn(*converted_inputs)
3167 return nest.pack_sequence_as(self._outputs_structure,
3168 [x.numpy() for x in outputs])

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
366 raise TypeError("Got two values for keyword '{}'.".format(unused_key))
367 raise TypeError("Keyword arguments {} unknown.".format(kwargs.keys()))
--> 368 return self._call_flat(args)
369
370 def _filtered_call(self, args, kwargs):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _call_flat(self, args)
431 # Only need to override the gradient in graph mode and when we have outputs.
432 if context.executing_eagerly() or not self.outputs:
--> 433 outputs = self._inference_function.call(ctx, args)
434 else:
435 if not self._gradient_name:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in call(self, ctx, args)
267 executing_eagerly=executing_eagerly,
268 config=function_call_options.config_proto_serialized,
--> 269 executor_type=function_call_options.executor_type)
270
271 if executing_eagerly:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/functional_ops.py in partitioned_call(args, f, tout, executing_eagerly, config, executor_type)
1081 outputs = gen_functional_ops.stateful_partitioned_call(
1082 args=args, Tout=tout, f=f, config_proto=config,
-> 1083 executor_type=executor_type)
1084 else:
1085 outputs = gen_functional_ops.partitioned_call(

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_functional_ops.py in stateful_partitioned_call(args, Tout, f, config, config_proto, executor_type, name)
495 else:
496 message = e.message
--> 497 _six.raise_from(_core._status_to_exception(e.code, message), None)
498 # Add nodes to the TensorFlow graph.
499 if not isinstance(Tout, (list, tuple)):

/usr/lib/python3/dist-packages/six.py in raise_from(value, from_value)

1 个答案:

答案 0 :(得分:0)

取消操作或步骤后,会引发CancelledError ,如您所见,here

实际上提出的解决方案here对我有用:我只是用model.predict(x) 替换了 model(x)。我仍然不知道为什么这样做。但是,我的情况涉及一个模型,在该模型中,我进行了带有手动定义的渐变的自定义操作。从thisthis之类的注释中我可以理解,渐变的变化可能与Op:StatefulPartitionedCall有关。

请注意,CancelledError在笔记本中似乎是持久的。一旦获得它,您需要重新启动内核,以使张量流再次正常工作。