Bottom line: Using the torch.set_num_threads() in a PyTorch program that has a Transformer module can significantly change the behavior of the program (in my case, for the better).

I was experimenting with a PyTorch program that uses a TransformerEncoder to do anomaly detection. See https://jamesmccaffreyblog.com/2022/07/25/testing-a-transformer-based-autoencoder-anomaly-detection-system/.

This program mysteriously stopped working one day.

During training I saw this:

Starting training
epoch =    0   loss = 1658.0013
epoch =   10   loss = 945.3817
epoch =   20   loss = 467.7127
epoch =   30   loss = 277.3138
epoch =   40   loss = 202.2976
epoch =   50   loss = 160.5351
epoch =   60   loss = 130.3890
epoch =   70   loss = 108.6009
epoch =   80   loss = 92.6008
epoch =   90   loss = 81.1246
Done

The loss value steadily decreased which indicated that the network containing the TransformerEncoder was learning. Good.

Then, one morning, the exact same program on the exact same machine started showing:

Starting training
epoch =    0   loss = 1658.0013
epoch =   10   loss = 945.3808
epoch =   20   loss = 955.5437
epoch =   30   loss = 950.6722
epoch =   40   loss = 949.4239
epoch =   50   loss = 956.1644
epoch =   60   loss = 954.2082
epoch =   70   loss = 951.4501
epoch =   80   loss = 945.4361
epoch =   90   loss = 954.1126
Done

The loss value immediately got stuck and so the network was not learning. This had me baffled. Somewhat unfortunately, all my machines belong to my company and are joined to the company network domain, which means that they are constantly being updated. I assumed that one of the updates had changed something.

A couple of days later, I ran into my work pal Ricky L who is an expert with transformer architecture. I described the weirdness in my system to him. He said he wasn’t surprised and that one thing for me to try was to set the number of threads explicitly with the statement torch.set_num_threads(1).

I looked up set_num_threads() in the PyTorch documentation and found exactly three sentences:

TORCH.SET_NUM_THREADS
torch.set_num_threads(int)
Sets the number of threads used for intraop parallelism on CPU.

That wasn’t too helpful so I just added a global call at the top of my program:

# uci_trans_anomaly.py

# Transformer based reconstruction error for UCI Digits
# PyTorch 1.10.0-CPU Anaconda3-2020.02  Python 3.7.6
# Windows 10/11 

import numpy as np
import torch as T

device = T.device('cpu') 
T.set_num_threads(1)  # I added this statement

# -----------------------------------------------------------

class Transformer_Net(T.nn.Module):
  . . . etc

And viola! The program was working correctly again.

Note: You can check how many threads your machine is using by default with the torch.get_num_threads() function.

I still don’t know the exact cause of the change in behavior of my program. But the moral of the story is that calling set_num_threads() in programs that use a Transformer module might be a good idea.

A ventriloquist and his/her dummy have two output streams but only one underlying thread of execution. Three old albums from the 1960s. Left: Geraldine and Ricky. Center: Happy Harry and Uncle Weldon. Right: Chris Kirby and Terry. Do not click on image to enlarge unless you’re prepared for several years of nightmares.