I wrote an article titled “Neural Anomaly Detection Using Keras” in the March 2019 issue of Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2019/03/01/neural-anomaly-detection-using-keras.aspx.

Anomaly detection, also called outlier detection, is the process of finding rare items in a dataset. Examples include finding fraudulent login events and fake news items. In general, anomaly detection is often extremely difficult, and there are many different techniques you can employ.

Demo run showing an anomalous ‘2’ data item

In my article, I demonstrate a neural network base approach. The idea is to create a neural autoencoder for a set of data. An autoencoder learns to predict its input, or, put another way, an autoencoder replicates data. The main principle of the technique presented in the article is that the more difficult a data item is to replicate, the more likely it is to be an anomaly.

The article uses the well-known MNIST image dataset. Each data item is 784 pixels representing a hand-drawn image of a digit from ‘0’ to ‘9’. For simplicity, instead of the entire 60,000 training items, I used just 1,000 images.

A typical, non-anomalous ‘2’ image.

So, there are several parts to the puzzle. First the raw data has to be prepared. Next you must define a neural autoencoder. I did so using the Keras code library which is a wrapper over the difficult-to-use TensorFlow library. Next you must define a metric that measures the difference/discrepancy between a predicted output and an actual output.

Anomaly detection using a deep neural autoencoder is not a well-known technique. An advantage of using a neural technique compared to a standard clustering technique is that neural techniques can handle non-numeric data by encoding that data. Most clustering techniques depend on a distance measure which means the source data must be strictly numeric.

Three images from an Internet search for “Dickens London”. Left: unknown setting and artist. Center: London Bridge under construction (Joseph Josiah Dodd). Right: unknown setting and artist but looks to me like it’s from a video game.