One of my standard neural network examples is image classification on the MNIST dataset. The full MNIST (modified National Institure of Standards and Technology) dataset has 60,000 images for training and 10,000 images for testing. Each image is a 28 x 28 (784 pixels) grayscale handwritten digit from ‘0’ to ‘9’. Each pixel value is an integer from 0 (white) to 255 (black).

I fetched the raw MNIST data from http://yann.lecun.com/exdb/mnist/. The data is stord in four .gz (gnu-zipped) files: train-images-idx3-ubyte.gz, train-labels-idx1-ubyte.gz, t10k-images-idx3-ubyte.gz, t10k-labels-idx1-ubyte.gz. I used the 7-Zip utility program to extract the four files. The data is stored in a proprietary binary format so I wrote a helper program to convert the binary data to text files. See https://jamesmccaffreyblog.com/2022/02/25/preparing-mnist-image-data-text-files-in-visual-studio-magazine/.

I used a 1,000-item subset of the training data, and a 100-item subset of the test data. After conversion to text, the data looks like:

0 0 0 . . 84 185 159 . . 5
0 0 0 . . 133 254 87 . . 9
0 0 0 . . 164 79 202 . . 7
. . .

Each line is one image. The first 784 values on each line are the pixel values. The last value on each line is the target digit, ‘0’ to ‘9’.

I designed a convolutional neural network that has two convolution layers, three linear layers, two pooling layers, and two dropout layers. The architecture was adapted from an example I found buried in the PyTorch documentation.

I used ReLU() activation on all layers except for the final layer where I used no activation (combined with CrossEntropyLoss() for training which automatically adds log_softmax() activation). I allowed the default Keras initialization — glorot_uniform() for weights and zeros() for biases.

For training, I used stochastic gradient descent optimization with a fixed learning rate of 0.05 and a batch size of 20.

The demo achieved 97.00% accuracy on the test data: not bad but it’s possible to do better by fiddling with the hyperparameters.

To use the trained model, just for fun, I created a fake image that sort of resembles a reversed ‘4’.

Note: I did the same problem using PyTorch 1.10.0 — see https://jamesmccaffreyblog.com/2022/05/26/mnist-image-classification-using-pytorch-1-10-on-windows-11/.

The Disney animated movie “Atlantis: The Lost Empire” (2001) was a box office flop. I had high hopes but even though the movie had excellent animation and art, it missed the mark on plot, dialog, and pacing. The animators created a complete Atlantean alphabet with digits — very nice. I would have loved for the movie to be a success but there’s always a chance for a new version or reboot.

Demo code. Replace “lt”, “gt”, “lte”, “gte” with Boolean operator symbols. My lame blog editor chokes on symbols.

# mnist_tfk.py
# MNIST using CNN and raw text data
# Keras 2.8.0 in TensorFlow 2.8.0 ("_tfk")
# Anaconda3-2020.02  Python 3.7.6  Windows 10/11

import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2'  # suppress CPU warn

import numpy as np
import tensorflow as tf
from tensorflow import keras as K
import matplotlib.pyplot as plt

# -----------------------------------------------------------

class MyLogger(K.callbacks.Callback):
  def __init__(self, n):
    self.n = n   # print loss & acc every n epochs

  def on_epoch_end(self, epoch, logs={}):
    if epoch % self.n == 0:
      curr_loss = logs.get('loss')
      curr_acc = logs.get('accuracy') * 100
      print("epoch = %4d  |  loss = %0.6f  |  acc = %0.2f%%" % \
(epoch, curr_loss, curr_acc))

# -----------------------------------------------------------

def main():
  # 0. get started
  print("\nBegin MNIST using Keras CNN and raw text data ")
  np.random.seed(1)
  tf.random.set_seed(1)

  # 1. load data
  # 784 tab-delim pixel values (0-255) then label (0-9)
  print("\nLoading 1000-train 100-test data from text file ")
  train_file = ".\\Data\\mnist_train_1000.txt" 
  all_train_xy = np.loadtxt(train_file, usecols=range(785),
      delimiter="\t", comments="#", dtype=np.float32)
  train_x = all_train_xy[:, 0:784]  # all rows, cols [0,783]
  train_x /= 255.0
  train_x = train_x.reshape(1_000, 28, 28, 1)

  train_y = all_train_xy[:, 784]
  train_y = K.utils.to_categorical(train_y, 10)

  test_file = ".\\Data\\mnist_test_100.txt" 
  all_test_xy = np.loadtxt(test_file, usecols=range(785),
      delimiter="\t", comments="#", dtype=np.float32)
  test_x = all_test_xy[:, 0:784]  # all rows, cols [0,783]
  test_x /= 255.0
  test_x = test_x.reshape(100, 28, 28, 1)

  test_y = all_test_xy[:, 784]
  test_y = K.utils.to_categorical(test_y, 10)

# -----------------------------------------------------------

  # 2. define model
  print("\nCreating CNN network with 2 conv and 3 linear ")
  # g_init = K.initializers.glorot_uniform(seed=1)
  
  x = K.layers.Input(shape=(28,28,1))
  con1 = K.layers.Conv2D(filters=32, kernel_size=(5,5), 
    activation='relu', padding='valid')(x)
  mp1 = K.layers.MaxPooling2D(pool_size=(2,2))(con1)
  do1 = K.layers.Dropout(0.25)(mp1)

  con2 = K.layers.Conv2D(filters=64, kernel_size=(5,5), 
    activation='relu', padding='valid')(do1)
  mp2 = K.layers.MaxPooling2D(pool_size=(2,2))(con2)

  # neural network phase
  z = K.layers.Flatten()(mp2)
  fc1 = K.layers.Dense(units=512, activation='relu')(z)
  do2 = K.layers.Dropout(0.50)(fc1)
  fc2 = K.layers.Dense(units=256, activation='relu')(do2)
    
  fc3 = K.layers.Dense(units=10, activation='softmax')(fc2)
 
  model = K.models.Model(x, fc3)

  lrn_rate = 0.05
  opt = K.optimizers.SGD(learning_rate=lrn_rate)
  # opt = K.optimizers.Adam(learning_rate=0.05)
  model.compile(loss='categorical_crossentropy',
    optimizer=opt, metrics=['accuracy'])

# -----------------------------------------------------------
  
  # 3. train model
  bat_size= 20
  max_epochs = 25
  print("\nbat_size = %3d " % bat_size)
  print("loss = categorical_crossentropy ")
  print("optimizer = SGD")
  print("lrn_rate = %0.3f " % lrn_rate)
  print("max_epochs = %3d " % max_epochs)


  print("\nStarting training")
  my_logger = MyLogger(n=5)  # progress every 5 epochs
  model.fit(train_x, train_y, batch_size=bat_size,
    epochs=max_epochs, verbose=0, callbacks=[my_logger])
  print("Done ")

# -----------------------------------------------------------

  # 4. evaluate model
  print("\nComputing model accuracy")
  eval = model.evaluate(train_x, train_y, verbose=0)
  acc_train = eval[1]
  print("Accuracy on training data = %0.4f" % acc_train)

  eval = model.evaluate(test_x, test_y, verbose=0)
  acc_test = eval[1]
  print("Accuracy on test data = %0.4f" % acc_test)

# -----------------------------------------------------------

  # 5. use model
  print("\nMaking prediction for fake image: ")
  x = np.zeros(shape=(28,28), dtype=np.float32)
  for row in range(5,23):
    x[row][9] = 180  # vertical line
  for rc in range(9,19):
    x[rc][rc] = 250  # diagonal
  for col in range(5,15):  
    x[14][col] = 200  # horizontal
  x /= 255.0

  plt.tight_layout()
  plt.imshow(x, cmap=plt.get_cmap('gray_r'))
  plt.show()

  x = x.reshape(1, 28, 28, 1)
  pred_probs = model.predict(x)  # sum to 1
  print("\nPrediction probabilities: ")
  np.set_printoptions(formatter={'float': '{: 0.4f}'.format})
  # np.set_printoptions(precision=4, suppress=True)
  print(pred_probs)
  
  digits = ['zero', 'one', 'two', 'three', 'four', 'five', 
    'six', 'seven', 'eight', 'nine' ]
  
  am = np.argmax(pred_probs)
  print("\nPredicted class is \'" + digits[am] + "\'")

# -----------------------------------------------------------

  # 6. save model
  print("\nSaving MNIST model to disk ")
  # mp = ".\\Models\\mnist_model.h5"
  # model.save(mp)

  print("\nEnd MNIST Keras CNN demo ")

if __name__ == "__main__":
  main()