In semi-supervised learning (SSL), you have a small set of normal training data with class labels, and a large set of data without class labels. Basically, you must use some algorithm to make intelligent guesses for the labels of the unlabeled data. and then use that data to train a model.

One of the long-term projects I’m working on is to explore new algorithms for semi-supervised learning. A significant challenge when working with semi-supervised learning is serving up data, both labeled and unlabeled.

For one specific semi-supervised algorithm, I needed to implement ways to serve up batches of labeled data and also batches of the same size of unlabeled data.

The idea is best explained by code snippets. I created some dummy labeled and unlabeled Employee data. The data looks like:

 1   0.24   1   0   0   0.2950   2
-1   0.39   0   0   1   0.5120   1
 1   0.63   0   1   0   0.7580   0
-1   0.36   1   0   0   0.4450   1
. . .

The first column is sex (-1 = male, 1 = female), the second column is age (normalized by dividing by 100), the third through fifth columns are city (three cities, one-hot encoded), the sixth column is income (normalized by dividing by $100,000). The seventh column is the label to predict — job type, one of three, ordinal encoded.

Here’s how my demo serves up training data in batches for the unlabeled data, using a standard PyTorch Dataset and DataLoader technique:

  bat_size = 2

  train_labeled_file = ".\\Data\\employee_train_labeled.txt"
  train_labeled_ds = \
    EmpLabeledDataset(train_labeled_file)  # 40 rows
  train_labeled_ldr = T.utils.data.DataLoader(train_labeled_ds,
    batch_size=bat_size, shuffle=True)

  print("\nFirst three bat_size = %d labeled items: " % \
    bat_size)
  for (bix, batch) in enumerate(train_labeled_ldr):
    X = batch[0]; y = batch[1]
    print(X); print(y)
    if bix == 2: break

But for the unlabeled data, my custom Dataset has a get_batch() method that returns randomly selected data. Note that the unlabeled data actually has labels but the labels aren’t used except to compute the accuracy of the semi-supervised learning algorithm.

  train_unlabeled_file = ".\\Data\\employee_train_unlabeled.txt"
  train_unlabeled_ds = \
    EmpUnlabeledDataset(train_unlabeled_file)  # 160 rows

  print("\nFirst three bat_size = %d unlabeled items: " % \
    bat_size)
  for i in range(3):
    batch = train_unlabeled_ds.get_batch(bat_size)
    X = batch[0]; y = batch[1]
    print(X); print(y)

The EmpUnlabeledDataset object is a hybrid in the sense that it can be consumed by a DataLoader object (useful for computing accuracy because it reads the class labels) or it can serve up batches of random data via the get_batch() method.

An enhancement would be to track which unlabeled items have been used and then will all have been used, reset by shuffling. This allows all unlabeled items to be used. As the code stands, there’s a chance that some items will be served up by get_batch() more often than other items.

I think the moral of the story is that in machine learning there are various levels of knowledge. A data scientist can be effective and productive by using only standard PyTorch techniques. But to explore new algorithms, it’s usually necessary to have a deeper level of knowledge so you can customize.

Movie posters are somewhat like class labels. Movie studios create official posters for each movie, but custom posters are possible too. Here are three nice custom posters for science fiction films that are distributed by the Criterion Collection company. Left: “The Atomic Submarine” (1960) – one of my favorite B-movie sci fi films. Center: “The Haunted Strangler” (1958) is essentially a Dr. Jekyll and Mr. Hyde story, starring Boris Karloff. Right: “First Man into Space” (1959) did not end well for the first man into space.

Demo code:

# employee_ssl_dataset.py
# datasets for semi-supervised learning
# predict job from sex, age, city, income
# PyTorch 1.10.0-CPU Anaconda3-2020.02  Python 3.7.6
# Windows 10 

import numpy as np
import torch as T
device = T.device('cpu')  # apply to Tensor or Module

# -----------------------------------------------------------

class EmpUnlabeledDataset(T.utils.data.Dataset):
  def __init__(self, src_file):
    all_xy = np.loadtxt(src_file, usecols=range(0,7),
      delimiter="\t", comments="#", dtype=np.float32)
    tmp_x = all_xy[:,0:6]   # cols [0,6) = [0,5]
    tmp_y = all_xy[:,6]     # 1-D
    self.x_data = T.tensor(tmp_x, dtype=T.float32)  #1-D
    self.y_data = T.tensor(tmp_y, dtype=T.int64)  # ignored 

    self.rnd = np.random.RandomState(1)
    self.n = len(self.x_data)
    self.indices = np.arange(self.n)
 
  def __len__(self):
    return self.n

  def __getitem__(self, idx):  # via DataLoader enumerate()
    preds = self.x_data[idx]
    trgts = self.y_data[idx] 
    sample = (preds, trgts)  # as a tuple
    return sample

  def get_batch(self, b_size):  # randomly selected
    self.rnd.shuffle(self.indices)
    preds = self.x_data[self.indices[0:b_size]]
    trgts = self.y_data[self.indices[0:b_size]]
    return (preds, trgts)  # as a tuple

# -----------------------------------------------------------

class EmpLabeledDataset(T.utils.data.Dataset):
  # sex age   city     income  job
  # -1  0.27  0  1  0  0.7610   2
  # +1  0.19  0  0  1  0.6550   0
  # sex: -1 = male, +1 = female
  # city: anaheim, boulder, concord
  # job: mgmt, supp, tech

  def __init__(self, src_file):
    all_xy = np.loadtxt(src_file, usecols=range(0,7),
      delimiter="\t", comments="#", dtype=np.float32)
    tmp_x = all_xy[:,0:6]   # cols [0,6) = [0,5]
    tmp_y = all_xy[:,6]     # 1-D
    self.x_data = T.tensor(tmp_x, dtype=T.float32)  #1-D
    self.y_data = T.tensor(tmp_y, dtype=T.int64) 

  def __len__(self):
    return len(self.x_data)

  def __getitem__(self, idx):  # automatic fetch
    preds = self.x_data[idx]
    trgts = self.y_data[idx] 
    sample = (preds, trgts)  # as a tuple
    return sample


# -----------------------------------------------------------

def main():
  # 0. get started
  print("\nBegin Employee semi-supervised datasets demo ")
  T.manual_seed(1)
  np.random.seed(1)
  
  # 1. create DataLoader objects
  print("\nCreating Employee labeled training dataset ")
  train_labeled_file = ".\\Data\\employee_train_labeled.txt"
  train_labeled_ds = \
    EmpLabeledDataset(train_labeled_file)  # 40 rows

  bat_size = 2
  train_labeled_ldr = T.utils.data.DataLoader(train_labeled_ds,
    batch_size=bat_size, shuffle=True)

  print("\nFirst three bat_size = %d labeled items: " % \
    bat_size)
  for (bix, batch) in enumerate(train_labeled_ldr):
    X = batch[0]; y = batch[1]
    print(X); print(y)
    if bix == 2: break

  print("\n=============================== ")

  print("\nCreating Employee unlabeled training dataset ")
  train_unlabeled_file = ".\\Data\\employee_train_unlabeled.txt"
  train_unlabeled_ds = \
    EmpUnlabeledDataset(train_unlabeled_file)  # 160 rows

  print("\nFirst three bat_size = %d unlabeled items: " % \
    bat_size)
  for i in range(3):
    batch = train_unlabeled_ds.get_batch(bat_size)
    X = batch[0]; y = batch[1]
    print(X); print(y)

  print("\nEnd Employee semi-supervised datasets demo ")

if __name__ == "__main__":
  main()