Question

我的数据集包含1000个用户的10天数据。我正在为每个用户培训和测试数据，以提高预测准确性。问题是，对于第一个用户，训练100个纪元需要花费5 secs，而对于第100个用户，训练100个纪元需要花费的时间超过five minutes。每个用户的培训时间都在增加。如何减少训练时间>由于位置点是分类的，因此实现了一种热编码来对位置点进行编码。

list = list_users[:100]
with open("accuracy_Lstm.csv","w") as f:
    f.write('user,LSTM \n')
    for user in list:
        user_data = newdataframe[newdataframe.user==user]
        encoded=encoding(user_data)
        X_train = []
        y_train = []
        for i in range(1, len(encoded)-96):
            X_train.append(encoded[i-1])
            y_train.append(encoded[i])
        X_train, y_train = np.array(X_train), np.array(y_train)
        X_test = encoded[-192:-96,:]
        X_true = encoded[-96:,:]
        X_trainL=X_train.reshape(X_train.shape[0],1,X_train.shape[1])
        time_steps = 1
    #Lstm
        model = Sequential()
        model.add(LSTM(X_train.shape[1], input_shape=(time_steps,X_train.shape[1]), activation='relu'))
        model.add(Dense(X_train.shape[1]))
        model.compile(loss='mse', optimizer='adam')
        model.fit(X_trainL, y_train, batch_size=96, epochs=100, verbose =1)
        model.summary()

        X_testL=X_test.reshape(X_test.shape[0],1,X_test.shape[1])

        pedL =one_hot_decode(model.predict(X_testL))
        true=one_hot_decode(X_true)
        try:
            accuracy = ((sum(x == y for x, y in zip(pedL, true)))/(len(pedL)))*100
        except ZeroDivisionError:
            accuracy = 0
        f.write(' %d,  %f \n'%(user, accuracy))

如何减少用户的培训时间？

Answer 1

问题是您在for循环的每次迭代中都在重新创建一个新模型。对于每个模型，这都是非常消耗内存的，应该避免。这就是为什么第一个模型训练得很快，而每个新模型却要慢一些的原因。即使每次迭代del model也无济于事，因为内存泄漏在张量流中。您可能会在for循环的开始时清除会话，但这本身很慢。

我建议您在循环之外创建模型，然后在每次迭代时重新初始化模型的权重（因为模型架构在两次迭代之间不会改变）

编辑：

如评论中所建议，以下是如何实现此想法的示例代码。由于问题本身不是有效的代码，因此我尚未测试并验证以下代码在没有内存泄漏的情况下工作，但是根据我以前的经验，我认为应该如此。

list = list_users[:100]

def get_model():
    model = Sequential()
    model.add(LSTM(X_train.shape[1], input_shape=(time_steps,X_train.shape[1]), activation='relu'))
    model.add(Dense(X_train.shape[1]))
    model.compile(loss='mse', optimizer='adam')
    model.summary()
    return(model, model.get_weights())

with open("accuracy_Lstm.csv","w") as f:
    f.write('user,LSTM \n')
    model, initial_weights = get_model()
    for user in list:
        user_data = newdataframe[newdataframe.user==user]
        encoded=encoding(user_data)
        X_train = []
        y_train = []
        for i in range(1, len(encoded)-96):
            X_train.append(encoded[i-1])
            y_train.append(encoded[i])
        X_train, y_train = np.array(X_train), np.array(y_train)
        X_test = encoded[-192:-96,:]
        X_true = encoded[-96:,:]
        X_trainL=X_train.reshape(X_train.shape[0],1,X_train.shape[1])
        time_steps = 1
    #Lstm
        model.set_weights(initial_weights)
        model.fit(X_trainL, y_train, batch_size=96, epochs=100, verbose =1)

        X_testL=X_test.reshape(X_test.shape[0],1,X_test.shape[1])

        pedL =one_hot_decode(model.predict(X_testL))
        true=one_hot_decode(X_true)
        try:
            accuracy = ((sum(x == y for x, y in zip(pedL, true)))/(len(pedL)))*100
        except ZeroDivisionError:
            accuracy = 0
        f.write(' %d,  %f \n'%(user, accuracy))

EDIT2：

由于每个用户的模型形状都在变化，因此您将需要在每次迭代时创建模型。因此上述解决方案将不起作用。您可以通过tf.keras.backend.clear_session()清除tensorflow内存，但是它很慢（这就是我在上述解决方案中尝试避免它的原因）。对于每个用户，以下解决方案可能会慢于5秒（由于增加了清除时间），但对于每个使用的解决方案，应保持不变，与您拥有的用户数无关。

list = list_users[:100]

def get_model(input_size):
    # clear the memory of tensorflow before creating a new model
    tf.keras.backend.clear_session()
    # creare new model
    model = Sequential()
    model.add(LSTM(input_size, input_shape=(time_steps,input_size), activation='relu'))
    model.add(Dense(input_size))
    model.compile(loss='mse', optimizer='adam')
    model.summary()
    return(model)

with open("accuracy_Lstm.csv","w") as f:
    f.write('user,LSTM \n')
    for user in list:
        # clear the memory of tensorflow at each new model
        tf.keras.backend.clear_session()

        user_data = newdataframe[newdataframe.user==user]
        encoded=encoding(user_data)
        X_train = []
        y_train = []
        for i in range(1, len(encoded)-96):
            X_train.append(encoded[i-1])
            y_train.append(encoded[i])
        X_train, y_train = np.array(X_train), np.array(y_train)
        X_test = encoded[-192:-96,:]
        X_true = encoded[-96:,:]
        X_trainL=X_train.reshape(X_train.shape[0],1,X_train.shape[1])
        time_steps = 1
    #Lstm
        model = get_model(X_train.shape[1])
        model.fit(X_trainL, y_train, batch_size=96, epochs=100, verbose =1)

        X_testL=X_test.reshape(X_test.shape[0],1,X_test.shape[1])

        pedL =one_hot_decode(model.predict(X_testL))
        true=one_hot_decode(X_true)
        try:
            accuracy = ((sum(x == y for x, y in zip(pedL, true)))/(len(pedL)))*100
        except ZeroDivisionError:
            accuracy = 0
        f.write(' %d,  %f \n'%(user, accuracy))

花时间获取训练数据。 LSTM

1 个答案: