Is there a way to directly read data from s3 for model training in tensorflow

时间:2018-04-04 09:03:26

标签: python tensorflow amazon-s3

I have tfrecords in s3 and I want to directly read those files and feed them into tensorflow training.

    filename = "s3location"
    filename_queue = tf.train.string_input_producer([filename])
    with tf.Session() as sess:
      _, value = tf.TFRecordReader().read(filename_queue)

      feature = {'train/image': tf.FixedLenFeature([], tf.string),
            'train/label': tf.FixedLenFeature([], tf.int64)}

      features = tf.parse_single_example(value, features=feature)


      image = tf.decode_raw(features['train/image'], tf.float32)


      label = tf.cast(features['train/label'], tf.int32)

      image = tf.reshape(image, [32, 32, 1])


      images, labels = tf.train.shuffle_batch([image, label], batch_size=50, capacity=100, num_threads=1, min_after_dequeue=10)

But when I am trying to feed this sess.run(train, feed_dict={x:images, y: labels, hold_prob: 0.5})

I get the following error - TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable feed values include Python scalars, strings, lists, numpy ndarrays, or TensorHandles.

I have 2 questions -

  1. Is there a way to read data directly from s3 in tensorflow. looks like the support was added,但我找不到任何示例。

  2. 如果支持,我的方法是否正确?

0 个答案:

没有答案