Bottom line: In early versions of PyTorch, you had to programmatically manipulate the gradients of tensors. But the torch.nn module eliminates much of the low level tensor manipulation you have to deal with.

A neural network has weights and biases that, along with a set of input values, determine the output value(s). Training is the process of adjusting the values of the weights and biases from initial random values to values that make the network produce correct output, based on training data with known correct outputs.

Each weight and bias has an associated gradient, which is a value that is used to adjust its associated weight to produce more accurate output. Computing gradients is difficult. One of the most significant features of PyTorch is the ability to automatically compute gradients.

In the early days of PyTorch you had to write quite a few statements to enable automatic computation of gradients. But the torch.nn module consists of wrapper code that eliminates much, but not all, of the gradient manipulation code you have to write.

I wrote a short demo to illustrate gradient manipulation code, which is now rarely necessary. The demo begins:

# gradients.py

import numpy as np
import torch as T

x = np.array([1, 2, 3], dtype=np.float32)
t1 = T.tensor(x, dtype=T.float32, requires_grad=True)
t2 = t1 * t1 + 1
t2.retain_grad()  # leaf 
z = t2.sum()

When you create a tensor, the default is that there is no associated gradient. If you want a gradient you must specify the requires_grad=true parameter, as shown for t1. However, initially the gradient is type None.

Tensor t2 gets a gradient because it’s created from operations on t1.

A scalar value z is computed from t2, because, somewhat unexpectedly “gradients can be implicitly created only for scalar outputs”.

At this point t1 and t2 form what’s called a computational graph, where t2 is a leaf node. In a non-demo scenario the computational graph would likely have several tensor nodes. Leaf nodes normally don’t get automatic computation of gradients unless you add a call to retain_grad() as is done for t2.

Next, the demo displays t1 and t2, and their gradients, and the function used to compute the gradients:

print("\nTensor t1 values, grad, grad_fn: ")
print(t1)
print(t1.grad)
print(t1.grad_fn)

print("\nTensor t2 values, grad, grad_fn: ")
print(t2)
print(t2.grad)
print(t2.grad_fn)

Tensor t1 has no gradient yet, and no gradient function because it’s not a computed tensor.

Tensor t2 has no gradient yet but it does have a gradient function because it’s a computed tensor.

The automatic calculation of the gradient values is triggered by the backward() statement on the scalar computed output:

print("\nExecuting backward()")
z.backward()

The demo concludes by displaying the new values for tensors t1 and t2:

 
print("\nTensor t1 values, grad, grad_fn: ")
print(t1)
print(t1.grad)
print(t1.grad_fn)

print("\nTensor t2 values, grad, grad_fn: ")
print(t2)
print(t2.grad)
print(t2.grad_fn)

The demo code essentially defines a function f(x) = x^2 + 1 where x is a tensor. The gradient of a function is the Calculus derivative so f'(x) = 2x. The demo sets x = (1, 2, 3) and so f(x) = x^2 + 1 = (2, 5, 10) and f'(x) = 2x = (2, 4, 6). You can see these values reflected in the t1 tensor.

Working with PyTorch gradients at a low level is quite difficult. The torch.nn module (developed in 2018) allows you to define a neural network where the tensors that define the network are automatically created with gradients.

In the very early days of PyTorch (before version 0.4) there were separate Tensor and Variable objects. A Tensor was intended to hold numeric values but without associated gradients. A Variable was intended to have associated gradients. (The complete actual differences between the old Tensor and Variable are a bit more complicated). The two objects were merged into a single Tensor object that could either have a gradient (requires_grad=True) or not (requires_grad=False). So, if you come across example PyTorch code that uses Variable, it’s old code and you should probably look for a better, newer example.

Technically, a gradient is a vector of Calculus partial derivatives. Three unexpected images returned from an Internet search for “gradient”. I have no idea why these three photos are related to gradient. Left: a bar called The Annex in Portland, OR. Center: Hollywood actress Penelope Cruz. Right: the Crab Nebula.