Suppose you have a dataset that looks like:

# age, income, politics
0.25, 0.67500,  2
0.41, 0.78000,  0
0.38, 0.59500,  1
0.32, 0.68000,  0
0.51, 0.49500,  2

You want to create a PyTorch multi-class classifier to predict political leaning (conservative = 0, moderate = 1, liberal = 2) from age and income.

If you use NLLLoss then you are good to go. The labels/targets are used as indices and so they must be integers, as above. But if you want to use MSELoss then the labels/targets must be one-hot encoded so that the data looks like this:

# age, income, politics
0.25, 0.67500,  0.0, 0,0, 1.0
0.41, 0.78000,  1.0, 0.0, 0.0
0.38, 0.59500,  0.0, 1.0, 0.0
0.32, 0.68000,  1.0, 0.0, 0.0
0.51, 0.49500,  0.0, 0.0, 1.0

You can manually encode the labels in a preprocessing step, or you can programmatically encode the labels when you read the data from file into memory, usually as a PyTorch Dataset object. Programmatically encoding the labels is generally preferable because then you won’t have to maintain two pairs of train and test files.

If you want to programmatically convert to one-hot, you can accept a system dependency and use an existing library, such as the scikit-learn sklearn.preprocessing.OneHotEncoder module, or you can write a custom program-defined function. Using a pre-built library saves time, but library functions are usually huge and complex because they have to anticipate every use case, and include scads of error checking.

When I worked in the product groups at my huge tech company, one of the biggest pains we dealt with was incompatibilities between system dependencies. This scarred me a bit and as a result I tend to have a huge aversion to system dependencies, and so I usually write program-defined functions when feasible. Bit note that “feasible” varies greatly from scenario to scenario.

Here’s a short and simple function to programmatically encode:

def vec_to_onehot(vec, n_classes):
  # vec is an array or r x 1, int64 or float32
  # result is r x n_classes
  vc = vec.reshape(-1).astype(np.int64)
  result = np.zeros((len(vc), n_classes), dtype=np.float32)
  for i in range(len(result)):
    result[i][vc[i]] = 1.0
  return result

The function could be called like this:

import numpy as np
import torch as T
device = T.device('cpu')
. . .

class DummyDataset(T.utils.data.Dataset):
  # age   income   politics
  # 0.25, 0.67500,  2

  def __init__(self, src_file):
    all_xy = np.loadtxt(src_file, usecols=range(0,3),
      delimiter=",", comments="#", dtype=np.float32)
    tmp_x = all_xy[:,0:2]   # cols [0,2) = [0,1]
    tmp_y = all_xy[:,2]     # 1D float32
    tmp_y = vec_to_onehot(tmp_y, n_classes=3) # 2D float32

    self.x_data = T.tensor(tmp_x,
      dtype=T.float32).to(device)
    self.y_data = T.tensor(tmp_y,
      dtype=T.float32).to(device) 
. . .

The labels data can be read into memory as either a 1D array or a 2D array. And the data can be read into memory as type int64 or float32. Therefore, the vec_to_onehot() function deals with all four possibilities by converting the input vector to a 1D array and casting the values to int64.

The code above reads all the data into a float32 2D array and then uses array slicing to extract the X matrix and the Y matrix. If the code had been:

tmp_y = all_xy[:,2:3]  # 2D float32
tmp_y = vec_to_onehot(tmp_y, n_classes=3) # 2D float32

Then, somewhat unexpectedly, before calling vec_to_onehot(), the tmp_y matrix is 2D instead of 1D. Details like this can drive you batty if you’re new to PyTorch.

Another option is to read the X and Y matrices separately:

tmp_x = np.loadtxt(src_file, usecols=range(0,2),
  delimiter=",", comments="#", dtype=np.float32)
tmp_y = np.loadtxt(src_file, usecols=2,
  delimiter=",", comments="#", dtype=np.int64) # 1D
tmp_y = vec_to_onehot(tmp_y, n_classes=3) # 2D float32

Here tmp_y is before calling vec_to_onehot(), the tmp_y matrix is 1D with int64 values. But if the call is:

tmp_y = np.loadtxt(src_file, usecols=range(2,3),
  delimiter=",", comments="#", dtype=np.int64) # 2D

Then tmp_y is a 2D matrix. Sheesh! Tricky stuff this.

The real moral of the story here is that deciding whether to use a built in library function or to use a program-defined function is a choice that depends on many factors, and experience is extremely valuable to make a good choice.

Implementing a custom function from scratch instead of using a library function has some risks. Cooking from scratch at home instead of buying from a store has some risks too.

Right: A Chucky meatloaf, where the bacon underwear is a super eleant touch.

Demo program:

# one_hot_demo.py

import numpy as np
import torch as T
device = T.device('cpu')

print("\nProgrammatic one-hot encoding demo ")

# -----------------------------------------------------------

def vec_to_onehot(vec, n_classes):
  # vec is an array or r x 1
  # result is r x n_classes
  vc = vec.reshape(-1).astype(np.int64)
  result = np.zeros((len(vc), n_classes), dtype=np.float32)
  for i in range(len(result)):
    result[i][vc[i]] = 1.0
  return result

# -----------------------------------------------------------

class DummyDataset(T.utils.data.Dataset):
  # age   income   politics
  # 0.25, 0.67500,  2

  def __init__(self, src_file):
    all_xy = np.loadtxt(src_file, usecols=range(0,3),
     delimiter=",", comments="#", dtype=np.float32)
    tmp_x = all_xy[:,0:2]   # cols [0,2) = [0,1]
    tmp_y = all_xy[:,2]     # 1D float32
    tmp_y = vec_to_onehot(tmp_y, n_classes=3) # 2D float32

    # tmp_x = np.loadtxt(src_file, usecols=range(0,2),
    #   delimiter=",", comments="#", dtype=np.float32)
    # tmp_y = np.loadtxt(src_file, usecols=2,
    #   delimiter=",", comments="#", dtype=np.int64)
    # tmp_y = vec_to_onehot(tmp_y, n_classes=3) # 2D float32

    self.x_data = T.tensor(tmp_x, dtype=T.float32).to(device)
    self.y_data = T.tensor(tmp_y, dtype=T.float32).to(device) 

  def __len__(self):
    return len(self.x_data)

  def __getitem__(self, idx):
    preds = self.x_data[idx]
    trgts = self.y_data[idx] 
    return preds, trgts  # as a Tuple

# -----------------------------------------------------------

print("\nCreating Dataset object and loading data ")
train_file = ".\\dummy_data.txt"
train_ds = DummyDataset(train_file)

print("Displaying loaded data ")
train_ldr = T.utils.data.DataLoader(train_ds,
  batch_size=5, shuffle=False)  # all 5 items
for (batch_idx, batch) in enumerate(train_ldr):
  X = batch[0]
  Y = batch[1]
  print("\nX = "); print(X)
  print("\nY = "); print(Y)

print("\nEnd demo ")