I contributed to an article titled “Researchers Demonstrate Transformer Architecture-Based Anomaly Detection for Cybersecurity” on the Pure AI web site. See https://pureai.com/articles/2022/08/02/ta-anomaly-detection.aspx.

Researchers at Microsoft have demonstrated a new technique for anomaly detection. The technique is based on deep neural transformer architecture (TA). TA is an architecture that was originally intended for natural language processing. However, over the past two years, TA based systems have been successfully adapted for other problem scenarios.

The screenshot below illustrates how the TA anomaly detection technique works. The system shown successfully identifies one anomalous item that had been placed into a dataset of 100 normal items. The data items are a subset of the UCI Digits dataset. Each item is a crude handwritten digit from “0” to “9.”

The experiment created an anomalous item using a technique called the fast gradient sign method (FGSM) attack.

TA was originally developed in 2017 and was designed to handle long sequences of words, such as a paragraph of text. TA systems proved to be very successful and quickly replaced earlier systems based on LSTM architecture (“long, short-term memory”).

The anomaly detection system scans each item in the source dataset and uses a Transformer component to generate a condensed latent representation of the item. Then, a standard deep neural network decodes the latent representations and expands each item back to a format that is the same as the source data. The detection system compares each item’s original value with its reconstructed value. Data items with large reconstruction error don’t fit the TA model and must be anomalous in some way.

I am quoted in the article: “We were somewhat surprised at how effective the transformer architecture anomaly detection system was on small dummy datasets. The TA system worked much better than other anomaly detection systems at detecting FGSM attack items.”

During World War II (1940-1945) almost all fighter aircraft had propellors in front. Experimental planes with a pusher configuration were an anomalous design. Here are three U.S. experiments, all introduced in 1943. Although promising, by 1943 it was clear that jet-powered aircraft were the future, so none of these designs were pursued. Left: The Vultee XP-54 “Swoose Goose”. Center: The Curtiss-Wright XP-55 “Ascender” (jokingly called the “ass ender”). Right: The Northrop XP-56 “Black Bullet”.