使不同程度的多项式拟合以使用自定义损失函数

Question

我试图将不同程度的多项式函数拟合到我生成的某些数据，并且在没有库的情况下进行梯度下降。我还使用了一个自定义损失函数，我手动计算了该函数的梯度（希望它是正确的）。问题是我得到残差平方的均值的无穷大和nan，我不知道我在做什么错。请帮助我。

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

#y(x) = sin(x) + noise
def F(x_in_rad):
    return np.sin(x_in_rad) + noise*np.random.normal(0,1,N)

noise = 0.5
N = 50
#X - N datapoints in radians
X = np.deg2rad(np.random.normal(0,1,N) * 359)
Y = F(X)


X = np.atleast_2d(X).T
Y = np.atleast_2d(Y).T
#split data
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33, random_state=42)

#mean of the squared residuals
def sq_res_mean(Y_pred,Y_real):
    return np.mean((Y_pred - Y_real)**2)

#create design matrix
def designmatpoly(X,degree):
    X = X[:,0]
    eye = np.ones(X.shape)
    rows = []
    rows.append(eye)
    for i in range(1,degree+1):
        rows.append(X**i)
    return np.stack(rows).T


#L2 norm squared; gradient = 2w
def C(w):
    return np.sum(w**2)


def gradientdescent(Amat, y, rate, numiter, lam, deg):
    n, p = Amat.shape
    whistory = []
    w_analytical = np.dot((np.dot(Amat.T,Amat) + lam*np.eye(deg+1, dtype=int))**(-1),np.dot(Amat.T,Y_train))
    losshistory = [] 
    #random weights initialized
    w = np.atleast_2d(np.random.randn(deg+1)).T
    for i in range(numiter): 
        loss = np.square(y - w[0] - np.dot(Amat, w)) + lam*C(w_analytical)
        whistory.append(w)
        losshistory.append(loss)
        grad = np.dot(-2*Amat.T, y - w[0] - Amat.dot(w)) + lam*2*w_analytical
        w = w - rate*grad
    return w, np.asarray(whistory), np.asarray(losshistory)

def model(degree,rate=0.0001, num_iters = 50, lam = 0.5):
    A_test =  designmatpoly(X_test,degree)
    A_train = designmatpoly(X_train,degree)
    wfin, whist, meanlosstrace = gradientdescent(A_train, Y_train, rate, num_iters, lam, degree)
    return wfin, A_test

degrees = []
sq_res_means = []
for i in range(1,10):
    wfin, A_test = model(degree=i)
    degrees.append(i)
    Y_pred = np.dot(A_test,wfin)
    sqrm = sq_res_mean(Y_pred,Y_test)
    sq_res_means.append(sqrm)
    print("deg",i,"sq_res_mean",sqrm)

损失功能：

Answer 1

我不确定，所有数字都是用稀疏变量名来回传递的，但是数字问题是，w向量在线性以上任何程度都呈指数螺旋形失控，直到溢出为止。这就是为什么您获得NaN值的原因。

从功能上讲，您计算出的梯度的大小与w向量成比例；即使学习率较低，也足以使w迅速陷入分歧。 rate*grad的规模仍然比w本身大。

我建议您将矩阵初始化为具有已知解决方案的简单系统，并在deg = 2处观看前2或3次迭代，然后看看如何超出您的期望。