For most PyTorch neural networks, you can use the built-in loss functions such as CrossEntropyLoss() and MSELoss() for training. But for some custom neural networks, such as Variational Autoencoders and Siamese Networks, you need a custom loss function.

The built-in loss functions return a small value when the two items being compared are close to each other, and a larger loss value when the two items are very different. A contrastive loss function is essentially two loss functions combined, where you specify if the two items being compared are supposed to be the same or if they’re supposed to be different. In pseudo-code:

def contrastive_loss(y1, y2, flag):
  if flag == 0:  # y1 y2 supposed to be same
    return small val if similar, large if diff
  else if flag == 1:  # y1 y2 supposed to be diff
    return small val if diff, large if similar

It is possible to define two different loss functions instead of one contrastive loss function but it’s standard practice to define just one function.

Note: In an Internet search for “contrastive loss”, I found several blog sites and a couple of research papers that present a completely different loss function that looks like:

I really don’t understand what’s going on. I suspect that this alternative loss function is used for different purposes and goes by a different name (which I can’t track down). This alternative definition of contrastive loss has one anchor item, one similar item, and many dissimilar items.

Update: After spending many, many hours scanning dozens of research papers, I now believe that the term “contrastive loss” originally meant the specific loss function defined in the 2006 research paper listed in the next paragraph, but over time, “contrastive loss” morphed into a generic term with several definitions, where the equation above is the most common. This is all related to an incredible explosion of research about unsupervised, semi-supervised, and self-supervised learning. The number of recently published papers on these topics is overwhelming.

Update: I found one research paper that calls this specific type of contrastive loss “normalized temperature-scaled cross entropy loss” and explored it using code. See https://jamesmccaffreyblog.com/2022/04/11/an-example-of-normalized-temperature-scaled-cross-entropy-loss/.

You can define a contrastive loss function however you like but most implementations use a definition derived from a 2006 research paper “Dimensionality Reduction by Learning an Invariant Mapping”. When I was scanning the Internet, I found some implementations that were good, some that were weak, and some that were just incorrect.

A screenshot of the relevant text from the research paper. The 1/2 terms in equation (4) are not conceptually correct if Y is always 0 or 1.

I implemented my own contrastive loss function for PyTorch. I used a class because all the built-in loss functions are classes, but a regular standalone function would work fine too. Using the research paper equation for loss function directly results in inefficient and hard-to-understand code, so I used the paper’s logic but not the equation. The research paper had a conceptual mistake by including a 1/2 factor, so I eliminated that term.

Code to test my contrastive loss function looks like:

  loss_func = ContrastiveLoss()  # instantiate a loss object

  y1 = T.tensor([[1.0, 2.0, 3.0],
                 [3.0, 4.0, 5.0]], dtype=T.float32).to(device)

  y2 = T.tensor([[1.0, 2.0, 3.0],
                 [3.0, 4.0, 5.0]], dtype=T.float32).to(device)

  loss = loss_func(y1, y2, 0)
  print(loss)  # 0.0 -- small; y1 y2 should be equal, they are

  loss = loss_func(y1, y2, 1)   
  print(loss)  # 4.0 -- large; y1 y2 should be different

Interesting. Good fun.

Black and white movies have high contrast that can enhance the nightmarish quality of scary films. Here are three of my favorite science fiction movies where contrastive cinematography was used to nice effect. Left: In “Godzilla” (1956) the scene where the creature first appears over an island hillside is terrifying. Center: In “Quatermass 2” (1957) alien parasites have taken over an English village and a large factory to use as their base of operations. Right: In “The Atomic Submarine” (1959) the crew of a U.S. submarine enters an underwater alien spaceship where the interior is pitch black except for an ominous walkway.

Demo code:

# constrastive_demo.py
# "Dimensionality Reduction by Learning an Invariant Mapping"

# PyTorch 1.10.0-CPU Anaconda3-2020.02  Python 3.7.6
# Windows 10 /11

import numpy as np
import torch as T
device = T.device('cpu')


# -----------------------------------------------------------

class ContrastiveLoss(T.nn.Module):
  def __init__(self, m=2.0):
    super(ContrastiveLoss, self).__init__()  # pre 3.3 syntax
    self.m = m  # margin or radius

  def forward(self, y1, y2, d=0):
    # d = 0 means y1 and y2 are supposed to be same
    # d = 1 means y1 and y2 are supposed to be different
    
    euc_dist = T.nn.functional.pairwise_distance(y1, y2)

    if d == 0:
      return T.mean(T.pow(euc_dist, 2))  # distance squared
    else:  # d == 1
      delta = self.m - euc_dist  # sort of reverse distance
      delta = T.clamp(delta, min=0.0, max=None)
      return T.mean(T.pow(delta, 2))  # mean over all rows
    
# -----------------------------------------------------------

def main():
  print("\nBegin contrastive loss demo \n")

  loss_func = ContrastiveLoss()

  y1 = T.tensor([[1.0, 2.0, 3.0],
                 [3.0, 4.0, 5.0]], dtype=T.float32).to(device)

  y2 = T.tensor([[1.0, 2.0, 3.0],
                 [3.0, 4.0, 5.0]], dtype=T.float32).to(device)

  y3 = T.tensor([[10.0, 20.0, 30.0],
                 [30.0, 40.0, 50.0]], dtype=T.float32).to(device)

  loss = loss_func(y1, y2, 0)
  print(loss)  # 0.0 -- small; y1 y2 should be equal

  loss = loss_func(y1, y2, 1)   
  print(loss)  # 4.0 -- large; y1 y2 should be different

  loss = loss_func(y1, y3, 0)  
  print(loss)  # 2591.99 -- large; y1 y3 should be equal

  loss = loss_func(y1, y3, 1)  
  print(loss)  # 0.0 -- small; y1 y2 should be different 

  print("\nEnd demo ")

if __name__ == "__main__":
  main()