I contributed technical content and some quotes to an article titled “Your Attention Please: Understanding AI Attention” on the Pure AI web site. See https://pureai.com/articles/2025/07/01/your-attention-please-understanding-ai-attention.aspx.

The key component of large language models (LLMs) such as GPT-x, is a software module called a Transformer. The key component of a Transformer is called an Attention module.

Suppose an input sequence is the sentence, “Watch the man code.” Briefly and loosely speaking, the Attention mechanism computes a vector of values for each of the four words in the input, where the values represent the words with contextual information that can be used to determine the strength of the relationship, or relevance, between the words. In this example, the word “the” is most closely associated to the word “man”, so the attention values for “the” and “man” can be used to compute a measure of their strong association.

There are many dozens of online articles that explain the Attention mechanism. But these explanations of Attention tend to be either 1.) at a very low level, dissecting the underlying code in great detail, or 2.) at a very high level, ignoring all the complex details.

The Pure AI article explains Attention at an intermediate level — low enough to provide useful information, but not so low as to overwhelm you with complex engineering details. The article is based on training materials that I developed while working at Microsoft.

Suppose an input sentence is, “Watch the man code.” The first step is to break the sentence down into separate words (technically, tokens). This process is called tokenization.

The next step is to convert the words/tokens into numeric vectors. For example, the word “watch” might be converted into [0.36, 1.67, . . 0.92]. This process is called word embedding. The idea is that an English word can have multiple meanings.

After embedding, the numeric vectors representing the words are augmented with values that indicate their position within the input sentence. This process is called positional encoding. The idea is that position is important.

I harnessed my minimal PowerPoint skills to create a diagram that illustrates how tokenization, word embedding, positional encoding, and attention are related.

After positional encoding, the numeric vectors representing the words and their position within the input sentence are sent to the Attention mechanism where there are converted into more complex vectors that have relevance information added. The idea is subtle.

The final result of the embedding, positional encoding, and attention process is a set of vector values that describe the source input sentence in a very thorough way.

The mechanics of Attention are actually simpler than they appear, but the underlying concepts are deeper than you might expect.

I give a few quotes:

The Pure AI editors asked Dr. James McCaffrey to comment. McCaffrey directs the data science and research efforts at Nukleus, a data analytics company located near Redmond, Washington. Before Nukleus, McCaffrey worked at Microsoft where he developed most of the content ideas presented in this article.

“The Attention mechanism is arguably one of the most important algorithms in the history of computer science, and having a basic understanding of Attention is usually essential for most people working with AI large language models,” he said.

Before the 1960s, movie opening credits (name of director, names of starring actors, names of producers, etc.) were usually just simple text information slapped up on the screen. But starting in the 1960s, movie opening credits (also called the title sequence) became fascinating art forms that called attention to themselves.

The opening credits were designed by Maurice Bender (1918-1991). Binder also created the famous Bond gun-barrel sequence.