As I write this blog post, one of the most active areas in machine learning research is semi-supervised learning, and the closely related self-supervised learning. These techniques use data with just a few labeled items (semi-supervised) or no labels at all (self-supervised).

A key component of semi-supervised and self-supervised techniques is a contrastive loss function that compares two positive items, relative to one or more negative items. The basic idea is that the loss between two similar items should be small and the loss between different items should be large.

The term “contrastive loss” is a generic term and there are many ways to implement a specific contrastive loss function. I encountered an interesting research paper titled “A Simple Framework for Contrastive Learning of Visual Representations” (2020), Chen, et al., where the authors define a contrastive loss function they call “normalized temperature-scaled cross entropy loss”.

First two pages of the research paper. The contrastive loss equation is in the green box and is also shown below.

The math definition of the loss function is short but somewhat intimidating.

I decided to dissect the function by doing a concrete example. First, there’s a sim(zi, zj) similarity function component. This is the normalized dot product of two vectors. My demo defines sim() as:

import numpy as np
def sim(v1, v2):
  v1_normed = v1 / np.linalg.norm(v1)
  v2_normed = v2 / np.linalg.norm(v2)
  return np.dot(v1_normed, v2_normed)  # normalized dot prod

My demo uses numpy vectors but the exact same ideas apply to PyTorch or TensorFlow tensors.

In the research equation, the Greek letter tau is a “temperature” meaning it’s just a numeric constant, such as 0.01, used to scale sim() values so they don’t become too large or too small.

My example sets up a batch of N=4 data items:

v1 = np.array([0.5, 0.6, 0.5, 0.6])
v2 = np.array([0.1, 0.1, 0.2, 0.2])
v3 = np.array([0.9, 0.8, 0.9, 0.8])
v4 = np.array([0.3, 0.7, 0.7, 0.3])

In a non-demo scenario, the source data items are usually pixel values of images. Those are sent to a neural network that (after several layers) generates so-called embeddings — abstract representations. Here I use a length of just 4 values for simplicity. In a non-demo scenario the embedding dimension would be much larger, perhaps 100 or 200.

Next, the demo creates 4 augmented versions of the real data items:

v5 = np.array([0.55, 0.65, 0.50, 0.60])  # from v1
v6 = np.array([0.10, 0.15, 0.25, 0.20])  # from v2
v7 = np.array([0.90, 0.85, 0.95, 0.80])  # from v3
v8 = np.array([0.35, 0.70, 0.75, 0.30])  # from v4

In a non-demo scenario, there are several well-known ways to slightly modify/mutate image data (Gaussian blur, random vertical flip, etc.) The augmented/mutated images are sent to the neural network to generate embeddings. Presumably, embeddings of augmented images will be similar to the embeddings of the corresponding source images. Instead of generating the mutated embeddings from augmented data, it’s possible to mutate the source embeddings directly. This allows you to work with non-numeric data that has been one-hot encoded. However, I haven’t seen this approach discussed anywhere.

My goal is to compute the contrastive loss for positive pair v1 and v5. Here v1 is the reference vector and v5 is the mutated version of v1. The loss equation needs the sim() of the reference vector v1 with all the other v(i), except not sim(v1,v1). I computed them as:

v1v1 = np.exp(sim(v1,v1)/tau)  # not used
v1v2 = np.exp(sim(v1,v2)/tau)
v1v3 = np.exp(sim(v1,v3)/tau)
v1v4 = np.exp(sim(v1,v4)/tau)
v1v5 = np.exp(sim(v1,v5)/tau) # should be small
v1v6 = np.exp(sim(v1,v6)/tau)
v1v7 = np.exp(sim(v1,v7)/tau)
v1v8 = np.exp(sim(v1,v8)/tau)

In a non-demo scenario you’d use a loop instead of manually computing each of the 2N-1 sim() values.

Finally, the contrastive loss for (v1, v5) is:

numerator = v1v5
denom = v1v2 + v1v3 + v1v4 + v1v5 + v1v6 + v1v7 + v1v8
loss_v1v5 = -np.log(numerator / denom)

print("\n%0.6f" % loss_v1v5)

This is just the contrastive loss for one positive pair: v1 and v5. The total loss for the batch would be the sum (or average) of the losses for all the positive pairs: (v1,v5), (v2,v6), (v3,v7), (v4,v8).

Fascinating stuff.

Three examples from an Internet image search for “contrastive photography”.

Demo program:

# contrastive_loss_demo.py

# "normalized temperature-scaled cross entropy loss"
# "A Simple Framework for Contrastive Learning 
#   of Visual Representations" (2020), Chen, et al.

import numpy as np

def sim(v1, v2):
  v1_normed = v1 / np.linalg.norm(v1)
  v2_normed = v2 / np.linalg.norm(v2)
  return np.dot(v1_normed, v2_normed)  # normalized dot prod

np.set_printoptions(precision=2)

print("\nDemo of normalized temp-scaled CE loss ")

# a batch of data
v1 = np.array([0.5, 0.6, 0.5, 0.6])
v2 = np.array([0.1, 0.1, 0.2, 0.2])
v3 = np.array([0.9, 0.8, 0.9, 0.8])
v4 = np.array([0.3, 0.7, 0.7, 0.3])

print("\nBatch of unlabeled data, v1 to v4: ")
print(v1); print(v2); print(v3); print(v4)

# augmented data
v5 = np.array([0.55, 0.65, 0.50, 0.60])  # from v1
v6 = np.array([0.10, 0.15, 0.25, 0.20])  # from v2
v7 = np.array([0.90, 0.85, 0.95, 0.80])  # from v3
v8 = np.array([0.35, 0.70, 0.75, 0.30])  # from v4

print("\nAugmented data, v5 to v8: ")
print(v5); print(v6); print(v7); print(v8)

tau = 0.10  # temperature

print("\nComputing loss for positive pair v1,v5 ")
# loss for positive pair (v1, v5)
v1v1 = np.exp(sim(v1,v1)/tau)  # not used
v1v2 = np.exp(sim(v1,v2)/tau)
v1v3 = np.exp(sim(v1,v3)/tau)
v1v4 = np.exp(sim(v1,v4)/tau)
v1v5 = np.exp(sim(v1,v5)/tau)  # should be small
v1v6 = np.exp(sim(v1,v6)/tau)
v1v7 = np.exp(sim(v1,v7)/tau)
v1v8 = np.exp(sim(v1,v8)/tau)

numerator = v1v5
denom = v1v2 + v1v3 + v1v4 + v1v5 + v1v6 + v1v7 + v1v8

loss_v1v5 = -np.log(numerator / denom)
print("\n%0.6f" % loss_v1v5)

print("\nEnd demo ")