我在LSTM模型中计算RMSE(均方根误差)时遇到了一些麻烦。该模型非常合适,并且我获得了良好的损失减少,但是当我尝试逆向转换我的yhat结果时,我收到以下错误:
non-broadcastable output operand with shape (399,1) doesn't match the broadcast shape (399,4)
这是我的代码:
预处理:
btc = pd.read_csv('live_bitcoin.csv')
twitter_sent = pd.read_csv('live_tweet.csv')
reddit_sent = pd.read_csv('live_reddit.csv')
btc.columns = ["price_usd","24h_volume_usd","market_cap_usd","available_supply","total_supply","percent_change_1h","percent_change_24h","percent_change_7d", "Sell", "Buy", "15m", "Stamp"]
twitter_sent.columns = ["Sentiment", "Stamp"]
reddit_sent.columns = ["Sentiment", "Stamp"]
merged = pd.merge(twitter_sent, btc, on='Stamp', how='inner').merge(reddit_sent, on='Stamp', how='inner')
data = merged[["Sentiment_x", "Sentiment_y","24h_volume_usd", "market_cap_usd", "available_supply","price_usd"]].groupby(merged['Stamp']).mean()
datag = data[["24h_volume_usd", "market_cap_usd", "available_supply","price_usd"]]
tw_sentiment = data["Sentiment_x"]
rdt_sentiment = data["Sentiment_y"]
print "Dataset size: " + str(len(datag))
print "Timespan: " + str(len(datag)/60) + " hours"
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(0, 1))
values = datag.values.reshape(-1, datag.shape[1])
tw_sentiment = tw_sentiment.values.reshape(-1, 1)
rdt_sentiment = rdt_sentiment.values.reshape(-1, 1)
tw_sentiment = tw_sentiment.astype('float32')
rdt_sentiment = rdt_sentiment.astype('float32')
values = values.astype('float32')
scaled = scaler.fit_transform(values)
训练:
train_size = int(len(scaled) * 0.7)
test_size = len(scaled) - train_size
train, test = scaled[0:train_size,:], scaled[train_size:len(scaled),:]
split = train_size
def create_dataset(dataset, look_back, tw_sentiment, rdt_sentiment, sent=False):
dataX, dataY = [], []
for i in range(len(dataset) - look_back):
if i >= look_back:
a = dataset[i-look_back:i+1, 0]
a = a.tolist()
if(sent==True):
current_tw_sentiment = tw_sentiment[i].tolist()[0]
current_rdt_sentiment = rdt_sentiment[i].tolist()[0]
a.append(current_tw_sentiment)
a.append(current_rdt_sentiment)
dataX.append(a)
dataY.append(dataset[i + look_back, 0])
print(len(dataY))
return np.array(dataX), np.array(dataY)
look_back = 2
trainX, trainY = create_dataset(train, look_back, tw_sentiment[0:train_size], rdt_sentiment[0:train_size], sent=True)
testX, testY = create_dataset(test, look_back, tw_sentiment[train_size:len(scaled)], rdt_sentiment[train_size:len(scaled)], sent=True)
trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
# Creating new model
model = Sequential()
model.add(LSTM(100, input_shape=(trainX.shape[1], trainX.shape[2]), return_sequences=True))
model.add(LSTM(100))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')
model.save('LSTM_14-03-2018.h5')
# Loading model
# model = load_model('models/LSTM_12-03-2018_GOOD.h5')
history = model.fit(trainX, trainY, epochs=300, batch_size=100, validation_data=(testX, testY), verbose=0, shuffle=False)
yhat = model.predict(testX)
yhat_inverse = scaler.inverse_transform(yhat.reshape(-1, 1))
testY_inverse = scaler.inverse_transform(testY.reshape(-1, 1))
rmse_sent = sqrt(mean_squared_error(testY_inverse, yhat_inverse))
print "Done"
print 'Test RMSE: %.3f' % rmse_sent
主要问题在于:
yhat_inverse = scaler.inverse_transform(yhat.reshape(-1,1))
testY_inverse = scaler.inverse_transform(testY.reshape(-1,1))
根据我的理解(仍然是ML中的初学者),我的yhat变量的形状为(399,1),因为我试图根据几个特征进行预测。我只想将我的数据恢复到之前的转换,因此RMSE错误以适当的比例返回。我基本上试图将价格重新调整到正常规模。在MinMaxScaler在预处理阶段完成数据后,我也永远不会重新转换数据。
任何可能出错的线索?
答案 0 :(得分:0)
假设您的数据集名为df
,并且您正在执行第(i)列的缩放,则必须看起来像这样:
df[list(df)[i]] = min_max_scaler.fit_transform(df[list(df)
[i]].values.reshape(-1,1))
答案 1 :(得分:0)
以下对我有用。
scaler2 = MinMaxScaler()
scaler2.min_, scaler2.scale_ = scaler.min_[0], scaler.scale_[0]
yhat_inverse = scaler2.inverse_transform(yhat)
testY_inverse = scaler2.inverse_transform(testY)