在大数据（> 30GB）上训练模型，得到“ tensorflow.GraphDef类型的协议缓冲区”错误

Question

我已经使用tensorflow建立了模型。我已经在大小为20MB的数据上测试了我们的模型。工作正常。但是，当我们在大小为34GB的数据上运行模型时，它开始抛出

的异常

  

tensorflow.python.framework.errors_impl.InvalidArgumentError：无法序列化tensorflow.GraphDef的协议缓冲区，因为序列化的大小（3161272441bytes）将大于限制（2147483647字节）

我已经用Google搜索上述异常。我找到了一种解决方案，在link中进行了描述，并相应地更改了train_input_fn函数

def train_input_fn(features, labels, batch_size):
    """An input function for training"""
    # Convert the inputs to a Dataset.
    assert features.values.shape[0] == labels.values.shape[0]
    #features = features.values
    #labels = labels.values
    features_placeholder = tf.placeholder(features.values.dtype, 
     features.values.shape)
    labels_placeholder = 
    tf.placeholder(labels.values.dtype,labels.values.shape)
    dataset = 
    tf.data.Dataset.from_tensor_slices((features_placeholder, 
    labels_placeholder))
    # Shuffle, repeat, and batch the examples.
    dataset = dataset.shuffle(1000).repeat().batch(batch_size)
    iterator = dataset.make_initializable_iterator()
    # Build the Iterator, and return the read end of the pipeline.
    el = iterator.get_next()
    with tf.Session() as sess:
        sess.run(iterator.initializer, feed_dict= 
         {features_placeholder: features.values,labels_placeholder: 
          labels.values})
          return el

但是现在我正在遵循异常

  

ValueError：功能应该是Tensor的字典。给定类型：<class 'tensorflow.python.framework.ops.Tensor'>

def train_input_fn(features, labels, batch_size):
   """An input function for training"""
   # Convert the inputs to a Dataset.
   dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
   # Shuffle, repeat, and batch the examples.
   dataset = dataset.shuffle(1000).repeat().batch(batch_size)
   # Build the Iterator, and return the read end of the pipeline.
   return dataset.make_one_shot_iterator().get_next()

my_feature_columns = []
for key in train_x.keys():
    vocabulary_feature_column =tf.feature_column.categorical_column_with_vocabulary_list(key=key,vocabulary_list=train_x[key].unique().tolist())
    embedding_column = tf.feature_column.embedding_column(categorical_column=vocabulary_feature_column,dimension=10)
    my_feature_columns.append(embedding_column)


classifier = tf.estimator.DNNClassifier(
        feature_columns=my_feature_columns,
        # Three hidden layers of 10 nodes.
        hidden_units=[30, 10],
        model_dir="/data/tfmodel",
        # The model must choose between 2 classes.
        n_classes=2)

classifier.train(input_fn=lambda:train_input_fn(X_train, y_train, 30),steps=2000)

如何在大型数据集上训练模型？