In a semi-supervised learning (SSL) classification problem, you have some data with labels and a lot more data without labels. Common SSL scenarios are medical data (a few dozens of people with a disease, many thousands with unknown status) and security data (a few dozens of data items that are known attacks, many thousands of items with unknown safety).

In ML research for SSL, a common setup is to use the MNIST image data where you use 1,000 of the 60,000 training images as-is with labels, but you artificially hide labels for the remaining 59,000 training images.

Note: Positive and unlabeled learning (PUL) is a particular kind of of semi-supervised learning with just two classes.

There are many different algorithms for SSL problems. They all involve guessing the labels for the unlabeled data in some way. There are two general SSL approaches. One approach is to 1.) guess the labels for all the unlabeled data up front, and then 2.) train a model using the labeled data and the unlabeled data with label guesses determined in phase 1.

Two pages (page 1 and 3) from the original research paper that describes the pseudo-label technique for semi-supervised learning.

A second approach for semi-supervised learning is to simultaneously guess labels for the unlabeled data and train the classification model. This second approach is sometimes called the pseudo-labels algorithm/technique.

The original paper that described the pseudo-label SSL technique is “Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks” (2013) by D. Lee. That paper is light on detail, and doesn’t provide source code, and so there are many possible ways to implement the algorithm. One approach is:

create model with random weight values

loop epoch in max_epochs
  compute an alpha based on epoch
  (initially 0, growing to 1)

  loop through all labeled data using DataLoader
    get a batch of labeled data
    (use existing labels)
    compute labeled_loss for curr model

    get a batch of unlabeled data picked randomly
    use curr model to compute pseudo labels
    compute unlabeled_loss for curr model
  
    # compute combined loss
    loss = labeled_loss + (alpha * unlabeled_loss)
    update model weights based on loss
  end-loop all labeled data
end-loop epoch

The key idea is the alpha factor. Early in the training iterations, alpha is 0 and so the combined loss depends only on the labeled data. This makes sense because at first, the label guesses for the unlabeled data are completely unreliable. Late in the training iterations, alpha will be 1 and so the combined loss will be 50% due to labeled data, and 50% due to unlabeled data (which presumably will be more reliable).

When I get some time, I’ll code up a demo. It’s one thing to think you understand an algorithm, but you can never be certain until you actually implement the algorithm and run a program.

In many cases, artists who are comissioned to create a book cover are not given very much information about the book they’re illustrating for, and so there are many artistic implementations possible.

Right: By 1973, “Fellowship” had become a world-wide phenomenon. This cover art was done by the author Tolkien. The art has an etherial aura about it.