Category Archives: PyTorch

One-Hot vs. One-Over-N-Hot Encoding for Neural Networks

Posted on August 15, 2025 by jamesdmccaffrey

Bottom line: I did an experiment where I used one-over-n-hot encoding instead of the standard one-hot encoding for categorical predictor variables for a neural network. The results were promising but not conclusive. Suppose you want to predict a person’s income … Continue reading →

Posted in Machine Learning, PyTorch | Leave a comment

Attention Regression Parameterized by Embedding Dim and Weights Dim

Posted on July 23, 2025 by jamesdmccaffrey

The goal of a machine learning regression problem is to predict a single numeric value. Classical ML regression techniques include linear regression, kernel ridge regression, gradient boosting regression, and others. I’ve been experimenting with a new algorithm for machine learning … Continue reading →

Posted in PyTorch, Transformers | Leave a comment

Adding a Transformer Module to a PyTorch Regression Network – Linear Layer Pseudo-Embedding and NLP Style Positional Encoding

Posted on July 9, 2025 by jamesdmccaffrey

I’ve been looking at adding a Transformer module to a PyTorch regression network. Because the key functionality of a Transformer is the attention mechanism, I’ve also been looking at adding a custom Attention module instead of a Transformer. There are … Continue reading →

Posted in PyTorch, Transformers | Leave a comment

How Large Language Model Attention is Related to Embedding and Positional Encoding

Posted on June 30, 2025 by jamesdmccaffrey

The key component of a large language model such as GPT-x, is a software module called a Transformer. The key component of a Transformer is called an Attention module. I was giving a talk about the large language model (LLM) … Continue reading →

Posted in Machine Learning, Transformers | Leave a comment

Adding a Transformer Module to a PyTorch Regression Network – Linear Layer Pseudo-Embedding

Posted on June 11, 2025 by jamesdmccaffrey

I’ve been looking at adding a Transformer module to a PyTorch regression network. Because the key functionality of a Transformer is the attention mechanism, I’ve also been looking at adding a custom Attention module instead of a Transformer. There are … Continue reading →

Posted in PyTorch, Transformers | Leave a comment

Running My Standard PyTorch Multi-Class Classification Demo on Colab from a MacOS Machine, 2025 Update

Posted on June 10, 2025 by jamesdmccaffrey

I sometimes present PyTorch training sessions at my workplace or at technical conferences. By far the biggest pain point is dealing with attendees’ PyTorch installations. So I’ve been investigating the feasibility of using the online Colab system. Briefly, Colab is … Continue reading →

Posted in PyTorch | Leave a comment

Running My Standard PyTorch Multi-Class Classification Demo on Colab from a Windows Machine, 2025 Update

Posted on June 2, 2025 by jamesdmccaffrey

When I work with PyTorch I almost always use a traditional local installation. Some of my colleagues like to use the Google colab online environment because you can run a PyTorch program in a browser on a machine that has … Continue reading →

Posted in PyTorch | Leave a comment

Adding a Transformer Module to a PyTorch Regression Network – No Numeric Pseudo-Embedding

Posted on May 28, 2025 by jamesdmccaffrey

I’ve been looking at adding a Transformer module to a PyTorch regression network. Because the key functionality of a Transformer is the attention mechanism, I’ve also been looking at adding a custom Attention module instead of a Transformer. There are … Continue reading →

Posted in PyTorch, Transformers | Leave a comment

Adding a Transformer Module to a PyTorch Regression Network – Classic NLP Positional Encoding

Posted on May 14, 2025 by jamesdmccaffrey

I’ve been looking at adding a Transformer module to a PyTorch regression network. Because the key functionality of a Transformer is the attention mechanism, I’ve also been looking at adding a custom Attention module instead of a Transformer. There are … Continue reading →

Posted in PyTorch, Transformers | 1 Comment

Adding a Transformer Module to a PyTorch Regression Network

Posted on April 30, 2025 by jamesdmccaffrey

I’ve been looking at adding a Transformer module to a PyTorch regression network. Because the key functionality of a Transformer is the attention mechanism, I’ve also been looking at adding a custom Attention module instead of a Transformer. For both … Continue reading →

Posted in PyTorch, Transformers | Leave a comment