I was looking at neural variational autoencoders (VAEs). A VAE is used to generate synthetic images, based on a set of real images. The code to create a VAR involves the Kullback-Leibler divergence (KLD) metric. KLD measures the difference between two different distributions, P and Q. The metric is a divergence rather than a distance because KLD(P,Q) does not equal KLD(Q,P) in general. If two distributions are the same, KLD = 0.

Compared to N(0,1), a Gaussian with mean = 1 and sd = 2 is moved to the right and is flatter. The KL divergence between the two distributions is 1.3069.

There is a special case of KLD when the two distributions being compared are Gaussian (bell-shaped) distributed. In such a situation, all you need to compute KLD(P,Q) is the means of the two distributions and their standard deviations (or equivalently their variances because var = sd^2).

And there’s a special, special case when P is Gaussian and Q is Gaussian with mean = 0 and standard deviation = 1. This special, special case is the one that occurs in the code implementation of a VAE. The source research paper gives the equation as kld = -0.5 * (1 + log(v) – (u * u) – v) where u is the mean and v is the variance of distribution P.

To make sure I understood all the nuances of calculating KLD for two Gaussian distributions, I wrote a Python demo program with three functions. The first computed KLD for the special, special case when P is Gaussian and Q is N(0,1). The second function computed KLD for any two Gaussians P and Q. The third function computed KLD for the special, special case, but using multidimensional PyTorch tensors.

For all three functions, I set P with mean = 1 and sd = 2, and Q with mean = 0 and sd = 1. All three functions returned a KLD metric value of 1.3069 which gave me confidence that my understanding of KLD for two Gaussian distributions was correct.

To summarize, Kullback-Leibler divergence measures how different two sets of numbers are. Computing KL divergence has special cases when the two sets of numbers are Gaussian distributed. KL divergence with Gaussians is used to implement a variational autoencoder (VAE). A variational autoencoder model is used to simulate real images.

Some movies use miniature building models like these to simulate real buildings. Left: This model of Hogwart’s school was used in every film of the Harry Potter series. Right: This is a model of the French National Museum of Natural History, located in Paris. C’est magnifique.

# kld_gaussians.py

import numpy as np
import torch as T

def kld_special(u, s):
  # special KL for N(u, s) and N(0,1)
  # https://arxiv.org/pdf/1312.6114.pdf
  v = s * s  # variance
  return -0.5 * (1 + np.log(v) - (u * u) - v)

def kld_gauss(u1, s1, u2, s2):
  # general KL two Gaussians
  # u2, s2 often N(0,1)
  # https://stats.stackexchange.com/questions/7440/ +
  # kl-divergence-between-two-univariate-gaussians
  # log(s2/s1) + [( s1^2 + (u1-u2)^2 ) / 2*s2^2] - 0.5
  v1 = s1 * s1
  v2 = s2 * s2
  a = np.log(s2/s1) 
  num = v1 + (u1 - u2)**2
  den = 2 * v2
  b = num / den
  return a + b - 0.5

def kld_tensors(u, logvar):
  kld = -0.5 * T.sum(1 + logvar - (u * u) - \
    logvar.exp())
  return kld

u = 1.0; s = 2.0
kld = kld_special(u, s)  # 1.3069
print("%0.4f" % kld)

u1 = 1.0; s1 = 2.0
u2 = 0.0; s2 = 1.0  # N(0,1) reference
kld = kld_gauss(u1, s1, u2, s2) 
print("%0.4f" % kld)

u = T.tensor([1.0], dtype=T.float32)
s = T.tensor([2.0], dtype=T.float32)
logvar = T.log(s*s)
kld = kld_tensors(u, logvar)
print(kld)