I’ve been looking at an interesting research paper titled “A Simple Framework for Contrastive Learning of Visual Representations” (2020) by T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. The main idea is to take unlabeled image data and use it to create an abstract representation (as in a numeric vector of, say, 500 values) of the images. This representation has no direct use, but it can be fine-tuned for downstream tasks. See my earlier post at https://jamesmccaffreyblog.com/2022/04/11/an-example-of-normalized-temperature-scaled-cross-entropy-loss/.

To help me understand, I refactored the architecture diagram in the paper. Here’s the diagram from the research paper:

And here’s my version:

My explanation:

The input is a 32 x 32 CIFAR-10 color image. The image is sent twice to a sequence of three augmentations: 1.) a random part of the image (such as a 9 x 9 block of pixels) is cropped and then resized back to 32 x 32 pixels, 2.) the colors are randomly distorted, 3.) Gaussian blur is applied. The result is a pair of augmented images, x1 and x2. Note that “augmented” usually means “added to” but in the context of contrastive loss, “augmented” means “mutated”.

The pair of augmented images (x1, x2) are fed to a ResNet-50 neural network. ResNet-50 is a large neural network with 50 layers used for image classification. Intermediate results of the ResNet-50 network (h1, h2), just after the average pooling layer, are fetched rather than the final output vector of 10 values. The (h1, h2) outputs from the ResNet-50 component are abstract representations of the two augmented images. These two abstract representations could be compared by a contrastive loss function. But it was discovered that passing the representations to a simple, single-hidden-layer neural network to get a pair derived representations (z1, z2) and then feeding the derived representations to the normalized temperature-scaled cross entropy contrastive loss function works better.

The results of the loss function are used to update all the weights and biases in the SimCLR system. This results in the internal h1 and h2 representations being better. After training, the internal h-representations can be used for downstream tasks.

SimCLR architecture is an example of what’s sometimes called a Siamese network. This is because you feed two inputs to the network — conceptually it’s like there are two identical networks that share the same weights. In addition to SimCLR, there are many other specific examples of Siamese networks, but they tend to be more complicated than SimCLR. In fact, the complexity of the other Siamese architectures is what motivated the creation of SimCLR (the “Sim” stands for “simple”).

One thing that both diagrams leave out is that a SimCLR network is trained using a batch of pairs of images. The first pair are similar to each other, but the other pairs are randomly selected and are assumed to be dissimilar. The similar and dissimilar pairs are actually fed to the contrastive loss function, not just the similar pair as shown.

Interesting stuff.

Images from an Internet search for “contrastive image”. I sort of understand photography but I don’t grok photography at a deep level.