MATLAB中的随机梯度下降算法

Question

我试图在MATLAB中实现随机梯度下降，但我在某处出错了。我认为也许我检查收敛的方式是不正确的（我不太确定如何在每次迭代时更新估算器），但我不确定。我一直试图拟合基本的线性数据，但我得到的结果相当遥远，我希望得到一些帮助。有人能够指出我出错的地方，以及为什么这不能正常工作？

谢谢！

以下是数据设置和一般代码：

clear all;
close all;
clc

N_features = 2;
d = 100;
m = 100;

X_train = 10*rand(d,1);
X_test = 10*rand(d,1);
X_train = [ones(d,1) X_train];
X_test = [ones(d,1) X_test];

y_train = 5 + X_train(:,2) + 0.5*randn(d,1);
y_test = 5 + X_test(:,2) + 0.5*randn(d,1);

gamma = 0.01; %learning rate

[sgd_est_train,sgd_est_test,SSE_train,SSE_test,w] = stoch_grad(d,m,N_features,X_train,y_train,X_test,y_test,gamma);

figure(1)
plot(X_train(:,2),sgd_est_train,'ro',X_train(:,2),y_train,'go')

figure(2)
plot(X_test(:,2),sgd_est_test,'bo',X_test(:,2),y_test,'go')

实际实现SGD的功能是：

% stochastic gradient descent

function [sgd_est_train,sgd_est_test,SSE_train,SSE_test,w] = stoch_grad(d,m,N_features,X_train,y_train,X_test,y_test,gamma)

    epsilon = 0.01; %convergence criterion
    max_iter = 10000;

    w0 = zeros(N_features,1); %initial guess
    w = zeros(N_features,1); %for convenience

    x = zeros(d,1);
    z = zeros(d,1);

    for jj=1:max_iter;
        for kk=1:d;
            x = X_train(kk,:)';
            z = gamma*((w0'*x-y_train(kk))*x);
            w = w0 - z;
        end

        if norm(w0-w,2)<epsilon
            break;
        else
            w0 = w;
        end
    end

    sgd_est_test = zeros(m,1);
    sgd_est_train = zeros(d,1);

    for ll=1:m;
        sgd_est_test(ll,1) = w'*X_test(ll,:)';
    end

    for ii=1:d;
        sgd_est_train(ii,1) = w'*X_train(ii,:)';
    end

    SSE_test = sum((sgd_est_test - y_test).^2);
    SSE_train = sum((sgd_est_train - y_train).^2);

end

Answer 1

我尝试将学习率降低到0.001，结果是：
这告诉我你的算法产生y = a x而不是y = a x + b形式的估计（由于某种原因忽略了常数项），你还需要降低学习率为了收敛。