When I was first learning how to use the PyTorch neural network library, I remember being confused by some of the example programs I found on the Internet. There seemed to be three different tanh() functions. After some investigation I discovered that yes, there are (at least) three different versions of tanh() in PyTorch.

Suppose you have a simple 4-7-3 neural network (for example, for the Iris dataset):

import torch as T

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(4, 7)
    self.oupt = T.nn.Linear(7, 3)

  def forward(self, x):
    z = T.tanh(self.hid1(x))
    z = T.log_softmax(self.oupt(z), dim=1)
    return z

The code uses the tanh() function that is part of the base torch library. Simple, effective, good. There’s also a different tanh() function in the torch.nn.functional module:

import torch as T

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(4, 7)
    self.oupt = T.nn.Linear(7, 3)

  def forward(self, x):
    z = T.nn.functional.tanh(self.hid1(x))
    z = T.log_softmax(self.oupt(z), dim=1)
    return z

Note: The torch.nn.functional.tanh() function has been deprecated but it still exists.

There’s also a Tanh() class:

import torch as T

class Net(T.nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.hid1 = T.nn.Linear(4, 7)
    self.oupt = T.nn.Linear(7, 3)
    self.my_tanh = T.nn.Tanh()

  def forward(self, x):
    z = self.my_tanh(self.hid1(x))
    z = T.log_softmax(self.oupt(z), dim=1)
    return z

The class version is a bit confusing. The Tanh() object has a built-in forward() method that calls a derived built-in __call__() method. The class can also be called directly:

  . . .
  z = T.nn.Tanh()(self.hid1(x))
  . . .

PyTorch has three different tanh() functions because, well, that’s the way Open Source software is. Open Source software evolves somewhat organically which leads to some inconsistencies, but overall, that’s a small price to pay.

Three different book covers for “Murder on the Orient Express” (1934) by Agatha Christie. One of my favorite mystery novels. The film version (1974) is one of my favorite mystery movies.