RuntimeWarning：梯度下降中在double_scalars中遇到溢出

Question

我正在尝试将梯度下降算法实现为线性回归。我认为已经弄清楚了数学部分，但是在Python中不起作用。

from sklearn.datasets import load_boston
import pandas as pd
import numpy as np
import random

data = load_boston()
df = pd.DataFrame(data['data'], columns=data['feature_names'])
y = data['target']
X = df.TAX

def RMSE(y, y_hat):
    return np.sqrt(sum((y - y_hat) ** 2) / len(y))

def partial_k(x, y, y_hat):
    n = len(y)
    gradient = 0
    for x_i, y_i, y_hat_i in zip(list(x), list(y), list(y_hat)):
        gradient += (y_i - y_hat_i) * x_i
    return -2 / n * gradient

def partial_b(y, y_hat):
    n = len(y)
    gradient = 0
    for y_i, y_hat_i in zip(list(y), list(y_hat)):
        gradient += (y_i - y_hat_i)
    return -2 / n * gradient

def gradient(X, y, n, alpha=0.01, loss=RMSE):
    loss_min = float('inf')

    k = random.random() * 200 - 100
    b = random.random() * 200 - 100

    for i in range(n):
        y_hat = k * X + b
        loss_new = loss(y, y_hat)
        if loss_new < loss_min:
            loss_min = loss_new
            print(f"round: {i}, k: {k}, b: {b}, {loss}: {loss_min}")
        k_gradient = partial_k(X, y, y_hat)
        b_gradient = partial_b(y, y_hat)
        k += -k_gradient * alpha
        b += -b_gradient * alpha
    return (k, b)
gradient(X, y, 200)

该脚本仅在第一次迭代中起作用，然后抛出警告；

/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:5: RuntimeWarning: overflow encountered in double_scalars
  """
/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:12: RuntimeWarning: overflow encountered in double_scalars
  if sys.path[0] == '':
/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:29: RuntimeWarning: invalid value encountered in double_scalars
/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:30: RuntimeWarning: invalid value encountered in double_scalars

Answer 1

看起来您的操作之一正在溢出类型。 What are the causes of overflow encountered in double_scalars besides division by zero?

如果您可以使用调试器运行代码，则可以找到导致溢出的行，并将类型更改为更大的值。

Answer 2

梯度下降的主要特征是将theta的值减小到最佳值，但是当alpha（学习率）较大时，梯度下降可能会超过最小值，并在图形上以振荡方式增加。为什么这里会发生溢出。

尝试降低alpha（学习率）或应用功能标准化。