Question

假设autograd处于打开状态（默认情况下为打开状态），那么执行以下操作之间是否有任何区别（除了缩进）？

with torch.no_grad():
    <code>

和

torch.set_grad_enabled(False)
<code>
torch.set_grad_enabled(True)

Answer 1

实际上不，问题使用的方式没有区别。当您查看no_grad的源代码时。您会看到它实际上是在使用torch.set_grad_enabled来存档此行为：

class no_grad(object):
    r"""Context-manager that disabled gradient calculation.

    Disabling gradient calculation is useful for inference, when you are sure
    that you will not call :meth:`Tensor.backward()`. It will reduce memory
    consumption for computations that would otherwise have `requires_grad=True`.
    In this mode, the result of every computation will have
    `requires_grad=False`, even when the inputs have `requires_grad=True`.

    Also functions as a decorator.


    Example::

        >>> x = torch.tensor([1], requires_grad=True)
        >>> with torch.no_grad():
        ...   y = x * 2
        >>> y.requires_grad
        False
        >>> @torch.no_grad()
        ... def doubler(x):
        ...     return x * 2
        >>> z = doubler(x)
        >>> z.requires_grad
        False
    """

    def __init__(self):
        self.prev = torch.is_grad_enabled()

    def __enter__(self):
        torch._C.set_grad_enabled(False)

    def __exit__(self, *args):
        torch.set_grad_enabled(self.prev)
        return False

    def __call__(self, func):
        @functools.wraps(func)
        def decorate_no_grad(*args, **kwargs):
            with self:
                return func(*args, **kwargs)
        return decorate_no_grad

在torch.set_grad_enabled语句中使用torch.no_grad而不是with时，还有>>> x = torch.tensor([1], requires_grad=True) >>> is_train = False >>> with torch.set_grad_enabled(is_train): ... y = x * 2 >>> y.requires_grad的附加功能，可让您控制打开或关闭梯度计算：

import torch
w = torch.rand(5, requires_grad=True)
print('Grad Before:', w.grad)
torch.set_grad_enabled(False)
with torch.enable_grad():
    scalar = w.sum()
    scalar.backward()
    # Gradient tracking will be enabled here.
torch.set_grad_enabled(True)

print('Grad After:', w.grad)

https://pytorch.org/docs/stable/_modules/torch/autograd/grad_mode.html

编辑：

@TomHale关于您的评论。我刚刚使用PyTorch 1.0进行了简短测试，结果发现渐变会处于活动状态：

Grad Before: None
Grad After: tensor([1., 1., 1., 1., 1.])

输出：

import torch
w = torch.rand(5, requires_grad=True)
print('Grad Before:', w.grad)
with torch.no_grad():
    with torch.enable_grad():
        # Gradient tracking IS enabled here.
        scalar = w.sum()
        scalar.backward()

print('Grad After:', w.grad)

因此将在此设置中计算梯度。

您在答案中发布的其他设置也会产生相同的结果：

Grad Before: None
Grad After: tensor([1., 1., 1., 1., 1.])

输出：

function giveitauniquename()
{
  so this is a comment
  echo "there's no need to further escape apostrophes/etc if you are commenting your code this way"
  the drawback is it will be stored in memory as a function as long as your script runs unless you explicitly unset it
  only valid-ish bash allowed inside for instance these would not work without the "pound" signs:
  1, for #((
  2, this #wouldn't work either
  function giveitadifferentuniquename()
  {
    echo nestable
  }
}

Answer 2

torch.autograd.enable_grad documentation说：

在no_grad上下文中启用梯度计算。在no_grad之外无效。

鉴于此措辞，预期如下：

torch.set_grad_enabled(False)
with torch.enable_grad:
    # Gradient tracking will NOT be enabled here.
torch.set_grad_enabled(True)

vs：

with torch.no_grad():
    with torch.enable_grad:
        # Gradient tracking IS enabled here.

但是与blue-phoenix shows一样，不是情况。

我提出了一个问题here。

PyTorch set_grad_enabled（False）与no_grad（）：

2 个答案: