Question

我已经创建了tensorflow hadoop环境https://github.com/tensorflow/examples/blob/master/community/en/docs/deploy/hadoop.md所需的所有环境。如我的Spark集群上的链接所示

我用了一个示例代码来训练模型并将模型h5py保存到HDFS中，并希望重新加载模型。

下面是我正在使用的示例代码

from keras.models import Sequential
from keras.models import load_model
import numpy
import os

seed = 7
numpy.random.seed(seed)

dataset = numpy.loadtxt("pima-indians-diabetes.csv", delimiter=",")

X = dataset[:,0:8]
Y = dataset[:,8]

model = Sequential()
model.add(Dense(12, input_dim=8, kernel_initializer='uniform', activation='relu'))
model.add(Dense(8, kernel_initializer='uniform', activation='relu'))
model.add(Dense(1, kernel_initializer='uniform', activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(X, Y, epochs=150, batch_size=10, verbose=0)

现在，我正在尝试将模型保存到本地目录，可以在其中保存和还原模型。

model.save("temp_model.h5") #### <----- it saved the model to local directory of my machine

del model

model = load_model("temp_model.h5") ##### <----- loaded the model file

当我尝试将其保存到HDFS

model.save("hdfs://default/tmp/mss/temp_model.h5")



Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/keras/engine/network.py", line 1090, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/usr/lib64/python2.7/site-packages/keras/engine/saving.py", line 379, in save_model
    f = h5dict(filepath, mode='w')
  File "/usr/lib64/python2.7/site-packages/keras/utils/io_utils.py", line 186, in __init__
    self.data = h5py.File(path, mode=mode)
  File "/usr/lib64/python2.7/site-packages/h5py/_hl/files.py", line 312, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
  File "/usr/lib64/python2.7/site-packages/h5py/_hl/files.py", line 148, in make_fid
    fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 98, in h5py.h5f.create
IOError: Unable to create file (unable to open file: name = 'hdfs://default/tmp/mss/TensorflowOnSpark/model_dir/model.h5', errno = 2, error message = 'No such file or directory', flags = 13, o_flags = 242)

现在要测试spark环境，我尝试在控制台上打印tensorflow HDFS设置所需的环境变量--->“ LD_LIBRARY_PATH”。

print("LD_LIBRARY_PATH : ",os.environ("LD_LIBRARY_PATH"))

>>>LD_LIBRARY_PATH : /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64

为什么我不能保存模型文件？

无法将张量流模型文件保存到HDFS

0 个答案: