LSTM。使用inverse_transform

时间:2018-03-14 10:41:47

标签: python machine-learning scikit-learn keras lstm

我在LSTM模型中计算RMSE(均方根误差)时遇到了一些麻烦。该模型非常合适,并且我获得了良好的损失减少,但是当我尝试逆向转换我的yhat结果时,我收到以下错误:

non-broadcastable output operand with shape (399,1) doesn't match the broadcast shape (399,4)

这是我的代码:

预处理:

btc = pd.read_csv('live_bitcoin.csv')
twitter_sent = pd.read_csv('live_tweet.csv')
reddit_sent = pd.read_csv('live_reddit.csv')

btc.columns = ["price_usd","24h_volume_usd","market_cap_usd","available_supply","total_supply","percent_change_1h","percent_change_24h","percent_change_7d", "Sell", "Buy", "15m", "Stamp"]
twitter_sent.columns = ["Sentiment", "Stamp"]
reddit_sent.columns = ["Sentiment", "Stamp"]

merged = pd.merge(twitter_sent, btc,  on='Stamp', how='inner').merge(reddit_sent, on='Stamp', how='inner')
data = merged[["Sentiment_x", "Sentiment_y","24h_volume_usd", "market_cap_usd", "available_supply","price_usd"]].groupby(merged['Stamp']).mean()
datag = data[["24h_volume_usd", "market_cap_usd", "available_supply","price_usd"]]
tw_sentiment = data["Sentiment_x"]
rdt_sentiment = data["Sentiment_y"]

print "Dataset size: " + str(len(datag))
print "Timespan: " + str(len(datag)/60) + " hours"

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(0, 1))
values = datag.values.reshape(-1, datag.shape[1])
tw_sentiment = tw_sentiment.values.reshape(-1, 1)
rdt_sentiment = rdt_sentiment.values.reshape(-1, 1)
tw_sentiment = tw_sentiment.astype('float32')
rdt_sentiment = rdt_sentiment.astype('float32')
values = values.astype('float32')
scaled = scaler.fit_transform(values)

训练:

train_size = int(len(scaled) * 0.7)
test_size = len(scaled) - train_size
train, test = scaled[0:train_size,:], scaled[train_size:len(scaled),:]
split = train_size

def create_dataset(dataset, look_back, tw_sentiment, rdt_sentiment, sent=False):
    dataX, dataY = [], []
    for i in range(len(dataset) - look_back):
        if i >= look_back:
            a = dataset[i-look_back:i+1, 0]
            a = a.tolist()
            if(sent==True):                
                current_tw_sentiment = tw_sentiment[i].tolist()[0]
                current_rdt_sentiment = rdt_sentiment[i].tolist()[0]
                a.append(current_tw_sentiment)
                a.append(current_rdt_sentiment)
            dataX.append(a)
            dataY.append(dataset[i + look_back, 0])
    print(len(dataY))
    return np.array(dataX), np.array(dataY)

look_back = 2
trainX, trainY = create_dataset(train, look_back, tw_sentiment[0:train_size], rdt_sentiment[0:train_size], sent=True)
testX, testY = create_dataset(test, look_back, tw_sentiment[train_size:len(scaled)], rdt_sentiment[train_size:len(scaled)], sent=True)

trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))

# Creating new model
model = Sequential()
model.add(LSTM(100, input_shape=(trainX.shape[1], trainX.shape[2]), return_sequences=True))
model.add(LSTM(100))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')
model.save('LSTM_14-03-2018.h5')

# Loading model
# model = load_model('models/LSTM_12-03-2018_GOOD.h5')

history = model.fit(trainX, trainY, epochs=300, batch_size=100, validation_data=(testX, testY), verbose=0, shuffle=False)
yhat = model.predict(testX)
yhat_inverse = scaler.inverse_transform(yhat.reshape(-1, 1))
testY_inverse = scaler.inverse_transform(testY.reshape(-1, 1))
rmse_sent = sqrt(mean_squared_error(testY_inverse, yhat_inverse))
print "Done"
print 'Test RMSE: %.3f' % rmse_sent

主要问题在于:

  yhat_inverse = scaler.inverse_transform(yhat.reshape(-1,1))
  testY_inverse = scaler.inverse_transform(testY.reshape(-1,1))

根据我的理解(仍然是ML中的初学者),我的yhat变量的形状为(399,1),因为我试图根据几个特征进行预测。我只想将我的数据恢复到之前的转换,因此RMSE错误以适当的比例返回。我基本上试图将价格重新调整到正常规模。在MinMaxScaler在预处理阶段完成数据后,我也永远不会重新转换数据。

任何可能出错的线索?

2 个答案:

答案 0 :(得分:0)

假设您的数据集名为df,并且您正在执行第(i)列的缩放,则必须看起来像这样:

df[list(df)[i]] = min_max_scaler.fit_transform(df[list(df) 
[i]].values.reshape(-1,1))

答案 1 :(得分:0)

以下对我有用。

scaler2 = MinMaxScaler() scaler2.min_, scaler2.scale_ = scaler.min_[0], scaler.scale_[0]

yhat_inverse = scaler2.inverse_transform(yhat) testY_inverse = scaler2.inverse_transform(testY)