One day I decided to test my knowledge of PyTorch gradients. With the early versions of PyTorch (before v0.4) you had to work with gradients at a low level. The torch.nn module greatly simplified working with gradients by doing a lot of the low-level work for you. So the challenge I posed to myself was to implement a logistic regression binary classifier using only tensors, without the torch.nn module.

I have implemented logisitc regression (LR) from scratch many times so I consider myself very knowledgeable about LR. Implementing LR with PyTorch without the torch.nn module was much more difficlt and time consuming than I thought it’d be, but I learned many details about working with PyTorch gradients.

I didn’t run into any one particular problem, it was a series of roughly a dozen medium hurdles. Some of the solutions to the hurdles involved reshape(), retain_grad(), and retain_graph().

With the availability of the torch.nn module, it’s not necessary to wrestle with PyTorch gradients for standard neural architectures. But the things I learned about PyTorch gradients will be useful the next time I have to implement a non-standard neural network

# logis_regress.py
# PyTorch logistic regression, no nn.Module
# demo to illustrate raw tensors

# PyTorch 1.5.0-CPU Anaconda3-2020.02  Python 3.7.6
# Windows 10 

import numpy as np
import torch as T
device = T.device("cpu")  # apply to Tensor or Module

# -----------------------------------------------------------

def forward(x, w, b):
 z = T.dot(x, w).reshape(1)
 z += b
 p = 1 / (1 + T.exp(-z))
 return p

def main():
  # 0. prepare
  print("\nBegin LR demo")
  T.manual_seed(1)
  np.random.seed(1)

  # 1. set up training data
  train_x = np.array([
    [1.5, 2.5], [3.5, 4.5], [6.5, 6.5],
    [4.5, 1.5], [5.5, 3.5], [7.5, 5.5]], dtype=np.float32)
  train_y = np.array([1, 1, 1, 0, 0, 0], dtype=np.long)

  print("\nTraining data:")
  print(train_x)
  print(train_y)

  train_x = T.tensor(train_x, dtype=T.float32).to(device)
  train_y = T.tensor(train_y, dtype=T.long).to(device)

  # 2. set up wts and biases
  lo = -0.01; hi = 0.01
  w = T.rand((2), dtype=T.float32,
    requires_grad=True).to(device)
  w = (hi - lo) * w + lo
  w.grad = T.zeros(2)
  w.retain_grad()

  b = T.zeros((1), dtype=T.float32,
    requires_grad=True).to(device)
  b.grad = T.zeros(1)
  b.retain_grad()

  # 3. train
  print("\nTraining model")
  lrn_rate = 0.05
  indices = np.arange(6)
  for epoch in range(0, 50):
    tot_loss = 0
    np.random.shuffle(indices)  
    for ii in range(len(indices)):
      i = indices[ii]
      x = train_x[i]
      target = train_y[i]
      oupt = forward(x, w, b)
 
      loss = (oupt - target).pow(2).sum()
      tot_loss += loss.item()

      loss.backward(retain_graph=True)  # compute gradients

      w.data += -1 * lrn_rate * w.grad.data
      b.data += -1 * lrn_rate * b.grad.data

      w.grad = T.zeros(2)
      b.grad = T.zeros(1)

    if epoch % 10 == 0:
      print("epoch = %4d " % epoch, end="")
      print("   loss = %6.4f" % (tot_loss / 6))

  print("Done")
  print("\nModel weights: ")
  print(w)
  print("Model bias: ")
  print(b)

  # 4. evaluate
  print("\nRunning model on train data:")
  for i in range(0,6):
    x = train_x[i]
    y = forward(x, w, b)
    print(y)

  # 5. predict
  unk = np.array([4.5, 4.5], dtype=np.float32)
  unk = T.tensor(unk, dtype=T.float32)
  p = forward(unk, w, b)
  print("\nPredicting for input = ")
  print(unk)
  print("\nPredicted class: ")
  print(p)

  print("\nEnd demo ")

if __name__ == "__main__":
  main()

Successful cooking, like successful coding, relies on attention to detail. But it’s more difficult to fix cooking errors than it is to fix coding errors.