I wrote an article titled “Preparing MNIST Image Data Text Files” in the February 2022 edition of Microsoft Visual Studio magazine. See https://visualstudiomagazine.com/articles/2022/02/01/preparing-mnist-image-data-text-files.aspx.

MNIST data is the Hello World of image classification. Most popular neural network libraries, including PyTorch, scikit and Keras, have some form of built-in MNIST dataset designed to work with the library. But there are two problems with using a built-in dataset. First, data access becomes a magic black box and important information is hidden. Second, the built-in datasets use all 60,000 training and 10,000 test images and these are very awkward to work with because they’re so large.

The article explains how to 1.) download the zipped binary source MNIST data files, 2.) extract the zipped binary data, 3.) convert the binary data to text data so that one line of data holds one MNIST image, and 4.) display the data to verify the conversion to text file has worked properly.

The MNIST images are grayscale for simplicity. Here are three photographs that use back-and-white for artistic effect.

Here’s the code for the demo program:

# converter_mnist.py
# Anaconda3-2020.02 - Python 3.7.6

import numpy as np
import matplotlib.pyplot as plt

# convert MNIST binary to text file; combine pixels and labels
# target format:
# pixel_1 (tab) pixel_2 (tab) . . pixel_784 (tab) digit

# 1. manually download four zipped-binary files from
#    yann.lecun.com/exdb/mnist/ 
# 2. use 7-Zip to unzip files, add ".bin" extension
# 3. determine format you want and modify script

def convert(img_file, label_file, txt_file, n_images):
  print("\nOpening binary pixels and labels files ")
  lbl_f = open(label_file, "rb")   # MNIST has labels (digits)
  img_f = open(img_file, "rb")     # and pixel vals separate
  print("Opening destination text file ")
  txt_f = open(txt_file, "w")      # output file to write to

  print("Discarding binary pixel and label files headers ")
  img_f.read(16)   # discard header info
  lbl_f.read(8)    # discard header info

  print("\nReading binary files, writing to text file ")
  print("Format: 784 pixel vals then label val, tab delimited ")
  for i in range(n_images):   # number images requested 
    lbl = ord(lbl_f.read(1))  # get label (unicode, one byte) 
    for j in range(784):  # get 784 vals from the image file
      val = ord(img_f.read(1))
      txt_f.write(str(val) + "\t") 
    txt_f.write(str(lbl) + "\n")
  img_f.close(); txt_f.close(); lbl_f.close()
  print("\nDone ")

def display_from_file(txt_file, idx):
  all_data = np.loadtxt(txt_file, delimiter="\t",
    usecols=range(0,785), dtype=np.int64)

  x_data = all_data[:,0:784]  # all rows, 784 cols
  y_data = all_data[:,784]    # all rows, last col

  label = y_data[idx]
  print("digit = ", str(label), "\n")

  pixels = x_data[idx]
  pixels = pixels.reshape((28,28))
  for i in range(28):
    for j in range(28):
      # print("%.2X" % pixels[i,j], end="")
      print("%3d" % pixels[i,j], end="")
      print(" ", end="")
    print("")

  plt.tight_layout()
  plt.imshow(pixels, cmap=plt.get_cmap('gray_r'))
  plt.show()  

# -----------------------------------------------------------

def main():
  n_images = 1000
  print("\nCreating %d MNIST train images from binary files " \
    % n_images)
  convert(".\\UnzippedBinary\\train-images.idx3-ubyte.bin",
          ".\\UnzippedBinary\\train-labels.idx1-ubyte.bin",
          "mnist_train_1000.txt", 1000)

  # n_images = 100
  # print("\nCreating %d MNIST test images from binary files " \
    % n_images)
  # convert(".\\UnzippedBinary\\t10k-images.idx3-ubyte.bin",
  #         ".\\UnzippedBinary\\t10k-labels.idx1-ubyte.bin",
  #         "mnist_test_100.txt", 100)

  print("\nShowing train image [0]: ")
  img_file = ".\\mnist_train_1000.txt"
  display_from_file(img_file, idx=0)  # first image
  
if __name__ == "__main__":
  main()

It wasn’t too long ago that MNIST data was considered a relatively large dataset, with 60,000 training images and 10,000 test images. But now MNIST classification is considered a very easy problem. That said, there still are many research papers that apply their ideas to MNIST data and so a complete knowledge of MNIST is pretty much a prerequisite for all data scientists and machine learning engineers.