Question

我正在尝试实现一个应该用来学习灰度图像的简单神经网络。输入由一个像素的2d索引组成，输出应为该像素的值。

网络的构造如下：每个神经元都连接到每个先前神经元的输入（即像素的索引）以及输出。输出只是该序列中最后一个神经元的输出。

这种网络在学习图像方面非常成功，例如here。

问题： 在我的实现中，损失函数保持在0.2和0.4之间，具体取决于神经元的数量，学习率和使用的迭代次数，这是非常糟糕的。同样，如果将输出与我们为此训练的输出进行比较，则看起来就像是噪音。但这是我第一次在网络中使用torch.cat，所以我不确定这是否是罪魁祸首。有人可以看到我在做什么错吗？

from typing import List
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.nn import Linear

class My_Net(nn.Module):
    lin: List[Linear]

    def __init__(self):
        super(My_Net, self).__init__()
        self.num_neurons = 10
        self.lin = nn.ModuleList([nn.Linear(k+2, 1) for k in range(self.num_neurons)])

    def forward(self, x):
        v = x
        recent = torch.Tensor(0)
        for k in range(self.num_neurons):
            recent = F.relu(self.lin[k](v))
            v = torch.cat([v, recent], dim=1)
        return recent

    def num_flat_features(self, x):
        size = x.size()[1:]
        num = 1
        for i in size():
            num *= i
        return num

my_net = My_Net()
print(my_net)

#define a small 3x3 image that the net is supposed to learn
my_image = [[1.0, 1.0, 1.0], [0.0, 1.0, 0.0], [0.0, 1.0, 0.0]] #represents a T-shape
my_image_flat = []    #output of the net is the value of a pixel
my_image_indices = [] #input to the net is are the 2d indices of a pixel
for i in range(len(my_image)):
    for j in range(len(my_image[i])):
        my_image_flat.append(my_image[i][j])
        my_image_indices.append([i, j])

#optimization loop
for i in range(100):
    inp = torch.Tensor(my_image_indices)

    out = my_net(inp)

    target = torch.Tensor(my_image_flat)
    criterion = nn.MSELoss()
    loss = criterion(out.view(-1), target)
    print(loss)

    my_net.zero_grad()
    loss.backward()
    optimizer = optim.SGD(my_net.parameters(), lr=0.001)
    optimizer.step()

print("output of current image")
print([[my_net(torch.Tensor([[i,j]])).item() for i in range(3)] for j in range(3)])
print("output of original image")
print(my_image)

Answer 1

是的，torch.cat是可能出现问题的。因此，您可以毫无问题地使用它。

这里的问题是，每次迭代都定义一个新的优化器。相反，您应该在定义模型后定义一次。

因此，进行了更改后，代码可以正常工作，并且损耗不断减少。我还每隔5000次迭代添加了打印输出，以显示进度。

from typing import List
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.nn import Linear

class My_Net(nn.Module):
    lin: List[Linear]

    def __init__(self):
        super(My_Net, self).__init__()
        self.num_neurons = 10
        self.lin = nn.ModuleList([nn.Linear(k+2, 1) for k in range(self.num_neurons)])

    def forward(self, x):
        v = x
        recent = torch.Tensor(0)
        for k in range(self.num_neurons):
            recent = F.relu(self.lin[k](v))
            v = torch.cat([v, recent], dim=1)
        return recent

    def num_flat_features(self, x):
        size = x.size()[1:]
        num = 1
        for i in size():
            num *= i
        return num

my_net = My_Net()
print(my_net)

optimizer = optim.SGD(my_net.parameters(), lr=0.001)



#define a small 3x3 image that the net is supposed to learn
my_image = [[1.0, 1.0, 1.0], [0.0, 1.0, 0.0], [0.0, 1.0, 0.0]] #represents a T-shape
my_image_flat = []    #output of the net is the value of a pixel
my_image_indices = [] #input to the net is are the 2d indices of a pixel
for i in range(len(my_image)):
    for j in range(len(my_image[i])):
        my_image_flat.append(my_image[i][j])
        my_image_indices.append([i, j])

#optimization loop
for i in range(50000):
    inp = torch.Tensor(my_image_indices)

    out = my_net(inp)

    target = torch.Tensor(my_image_flat)
    criterion = nn.MSELoss()
    loss = criterion(out.view(-1), target)
    if i % 5000 == 0:
        print('Iteration:', i, 'Loss:', loss)

    my_net.zero_grad()
    loss.backward()
    optimizer.step()
print('Iteration:', i, 'Loss:', loss)

print("output of current image")
print([[my_net(torch.Tensor([[i,j]])).item() for i in range(3)] for j in range(3)])
print("output of original image")
print(my_image)

丢失的输出：

Iteration: 0 Loss: tensor(0.4070)
Iteration: 5000 Loss: tensor(0.1315)
Iteration: 10000 Loss: tensor(1.00000e-02 *
       8.8275)
Iteration: 15000 Loss: tensor(1.00000e-02 *
       5.6190)
Iteration: 20000 Loss: tensor(1.00000e-02 *
       3.2540)
Iteration: 25000 Loss: tensor(1.00000e-02 *
       1.3628)
Iteration: 30000 Loss: tensor(1.00000e-03 *
       4.4690)
Iteration: 35000 Loss: tensor(1.00000e-03 *
       1.3582)
Iteration: 40000 Loss: tensor(1.00000e-04 *
       3.4776)
Iteration: 45000 Loss: tensor(1.00000e-05 *
       7.9518)
Iteration: 49999 Loss: tensor(1.00000e-05 *
       1.7160)

因此，在这种情况下，损失下降到0.000017。我必须承认，您的错误表面确实参差不齐。取决于初始权重，它也可能收敛到0.17，0.10等的最小值。等等。收敛的局部最小值可能非常不同。因此，您可以尝试在较小的范围内初始化权重。

顺便说一句。这是不更改定义优化器位置的输出：

Iteration: 0 Loss: tensor(0.5574)
Iteration: 5000 Loss: tensor(0.5556)
Iteration: 10000 Loss: tensor(0.5556)
Iteration: 15000 Loss: tensor(0.5556)
Iteration: 20000 Loss: tensor(0.5556)
Iteration: 25000 Loss: tensor(0.5556)
Iteration: 30000 Loss: tensor(0.5556)
Iteration: 35000 Loss: tensor(0.5556)
Iteration: 40000 Loss: tensor(0.5556)
Iteration: 45000 Loss: tensor(0.5556)

pytorch的autograd可以处理torch.cat吗？

1 个答案: