Every few months I revisit my standard neural network examples to make sure that changes in the underlying code libraries (PyTorch, Keras/TensorFlow) haven’t introduced a breaking change(s). One of my standard examples is autoencoder anomaly detection.
The idea is to take a set of data and implement a deep neural network that predicts its input. The values of the interior hidden layer of nodes is a condensed representation of the input. The output nodes are a reconstruction of the input. Data items where the reconstructed input is very different from the associated input are anomalous in some way.
My demo uses a synthetic set of Employee data. There are x feature variables: employee sex (M, F), age, city (anaheim, boulder, concord), annual income, and job-type (mgmt, supp, tech). There are 240 items. The normalized and encoded data looks like:
# sex age city income job_type -1 0.27 0 1 0 0.7610 0 0 1 1 0.19 0 0 1 0.6550 0 1 0 . . .
My demo network uses a 9-4-(2)-4-9 architecture. The input and output size are determined by the data, but the number of hidden layers and the number of nodes in each, are hyperparameters that must be determined by trial and error. The middle hidden layer, with 2 nodes, represents a condensed version of a data item. This internal representation isn’t used directly — it’s used to reconstruct a data item. The difference between a 9-value input item and its 9-value output is used to find anomalies.
A problem facing artists who want to create images of alien animals and vegetation is to find a balance between images that are too anomalous to real plants and animals (making them look implausible) and images that are not anomalous enough (making them look not alien enough). Artist Jorge Abalo creates beautiful digital renderings of alien plants — a perfect balance of realistic and anomalous to my eye.
Demo code. Replace “lt”, “gt”, “let”, “gte” with Boolean operator symbols — my lame blog editor chokes on symbols.
# employee_autoanom_tfk.py
# autoencoder reconstruction error anomaly detection
# Keras 2.8.0 in TensorFlow 2.8.0 ("_tfk")
# Anaconda3-2020.02 Python 3.7.6 Windows 10/11
import os
os.environ['TF_CPP_MIN_LOG_LEVEL']='2' # suppress CPU warn
import numpy as np
import tensorflow as tf
from tensorflow import keras as K
# -----------------------------------------------------------
class MyLogger(K.callbacks.Callback):
def __init__(self, n):
self.n = n # print loss every n epochs
# self.data_x = data_x # for accuracy
# self.data_y = data_y
def on_epoch_end(self, epoch, logs={}):
if epoch % self.n == 0:
curr_loss = logs.get('loss') # loss on curr batch
print("epoch = %4d | loss = %0.6f " % \
(epoch, curr_loss))
# -----------------------------------------------------------
def analyze_error(model, data_x):
largest_err = 0.0
worst_x = None
worst_y = None
n_features = len(data_x[0]) # 9 predictors
for i in range(len(data_x)):
X = data_x[i].reshape(1,-1)
Y = model(X)
err = tf.reduce_sum( (X-Y)*(X-Y) ) # across all predictors
err /= n_features
if err "gt"For largest_err:
largest_err = err
worst_x = X
worst_y = Y.numpy()
np.set_printoptions(formatter={'float': '{: 0.4f}'.format})
print("Largest reconstruction error: %0.4f" % largest_err)
print("Worst data item = ")
print(worst_x)
print("Its reconstruction = " )
print(worst_y)
# -----------------------------------------------------------
def main():
# 0. prepare
print("\nBegin Employee autoencoder anomaly using Keras ")
np.random.random(1)
tf.random.set_seed(1)
# 1. load data
# sex age city income job_type
# -1 0.27 0 1 0 0.7610 0 0 1
# +1 0.19 0 0 1 0.6550 1 0 0
print("\nLoading Employee data into memory ")
data_file = ".\\Data\\employee_all.txt" # 240 lines
data_x = np.loadtxt(data_file, usecols=[0,1,2,3,4,5,6,7,8],
delimiter="\t", comments="#", dtype=np.float32)
# -----------------------------------------------------------
# 2. create network
print("\nCreating 9-4-(2)-4-9 network ")
model = K.models.Sequential()
model.add(K.layers.Dense(units=4, input_dim=9,
activation='tanh', kernel_initializer='glorot_uniform',
bias_initializer='zeros')) # enc1
model.add(K.layers.Dense(units=2,
activation='tanh', kernel_initializer='glorot_uniform',
bias_initializer='zeros')) # enc2
model.add(K.layers.Dense(units=4,
activation='tanh', kernel_initializer='glorot_uniform',
bias_initializer='zeros')) # dec1
model.add(K.layers.Dense(units=9,
activation=None, kernel_initializer='glorot_uniform',
bias_initializer='zeros')) # dec2
opt = K.optimizers.Adam(learning_rate=0.005)
model.compile(loss='mean_squared_error',
optimizer=opt, metrics=['mse'])
# -----------------------------------------------------------
# 3. train model
print("\nbat_size = 10 ")
print("loss = mean_squared_error ")
print("optimizer = Adam ")
print("lrn_rate = 0.005 ")
my_logger = MyLogger(100)
print("\nStarting training ")
h = model.fit(data_x, data_x, batch_size=10,
epochs=1000, verbose=0, callbacks=[my_logger])
print("Done ")
# -----------------------------------------------------------
# 4. find item with largest reconstruction error
print("\nAnalyzing data for largest reconstruction error \n")
analyze_error(model, data_x)
# -----------------------------------------------------------
# 5. save model
# print("\nSaving trained model ")
# model.save_weights(".\\Models\\employee_model_wts.h5")
# model.save(".\\Models\\employee_model.h5")
# -----------------------------------------------------------
print("\nEnd Employee autoencoder anomaly demo ")
if __name__=="__main__":
main()
Demo data:
# employee_all.txt # sex (M = -1, F = +1), age / 100, # city (anaheim = 100, boulder = 010, concord = 001), # income / 100_00, # job_type (mgmt = 100, supp = 010, tech = 001) # 1 0.24 1 0 0 0.2950 0 0 1 -1 0.39 0 0 1 0.5120 0 1 0 1 0.63 0 1 0 0.7580 1 0 0 -1 0.36 1 0 0 0.4450 0 1 0 1 0.27 0 1 0 0.2860 0 0 1 1 0.50 0 1 0 0.5650 0 1 0 1 0.50 0 0 1 0.5500 0 1 0 -1 0.19 0 0 1 0.3270 1 0 0 1 0.22 0 1 0 0.2770 0 1 0 -1 0.39 0 0 1 0.4710 0 0 1 1 0.34 1 0 0 0.3940 0 1 0 -1 0.22 1 0 0 0.3350 1 0 0 1 0.35 0 0 1 0.3520 0 0 1 -1 0.33 0 1 0 0.4640 0 1 0 1 0.45 0 1 0 0.5410 0 1 0 1 0.42 0 1 0 0.5070 0 1 0 -1 0.33 0 1 0 0.4680 0 1 0 1 0.25 0 0 1 0.3000 0 1 0 -1 0.31 0 1 0 0.4640 1 0 0 1 0.27 1 0 0 0.3250 0 0 1 1 0.48 1 0 0 0.5400 0 1 0 -1 0.64 0 1 0 0.7130 0 0 1 1 0.61 0 1 0 0.7240 1 0 0 1 0.54 0 0 1 0.6100 1 0 0 1 0.29 1 0 0 0.3630 1 0 0 1 0.50 0 0 1 0.5500 0 1 0 1 0.55 0 0 1 0.6250 1 0 0 1 0.40 1 0 0 0.5240 1 0 0 1 0.22 1 0 0 0.2360 0 0 1 1 0.68 0 1 0 0.7840 1 0 0 -1 0.60 1 0 0 0.7170 0 0 1 -1 0.34 0 0 1 0.4650 0 1 0 -1 0.25 0 0 1 0.3710 1 0 0 -1 0.31 0 1 0 0.4890 0 1 0 1 0.43 0 0 1 0.4800 0 1 0 1 0.58 0 1 0 0.6540 0 0 1 -1 0.55 0 1 0 0.6070 0 0 1 -1 0.43 0 1 0 0.5110 0 1 0 -1 0.43 0 0 1 0.5320 0 1 0 -1 0.21 1 0 0 0.3720 1 0 0 1 0.55 0 0 1 0.6460 1 0 0 1 0.64 0 1 0 0.7480 1 0 0 -1 0.41 1 0 0 0.5880 0 1 0 1 0.64 0 0 1 0.7270 1 0 0 -1 0.56 0 0 1 0.6660 0 0 1 1 0.31 0 0 1 0.3600 0 1 0 -1 0.65 0 0 1 0.7010 0 0 1 1 0.55 0 0 1 0.6430 1 0 0 -1 0.25 1 0 0 0.4030 1 0 0 1 0.46 0 0 1 0.5100 0 1 0 -1 0.36 1 0 0 0.5350 1 0 0 1 0.52 0 1 0 0.5810 0 1 0 1 0.61 0 0 1 0.6790 1 0 0 1 0.57 0 0 1 0.6570 1 0 0 -1 0.46 0 1 0 0.5260 0 1 0 -1 0.62 1 0 0 0.6680 0 0 1 1 0.55 0 0 1 0.6270 1 0 0 -1 0.22 0 0 1 0.2770 0 1 0 -1 0.50 1 0 0 0.6290 1 0 0 -1 0.32 0 1 0 0.4180 0 1 0 -1 0.21 0 0 1 0.3560 1 0 0 1 0.44 0 1 0 0.5200 0 1 0 1 0.46 0 1 0 0.5170 0 1 0 1 0.62 0 1 0 0.6970 1 0 0 1 0.57 0 1 0 0.6640 1 0 0 -1 0.67 0 0 1 0.7580 0 0 1 1 0.29 1 0 0 0.3430 0 0 1 1 0.53 1 0 0 0.6010 1 0 0 -1 0.44 1 0 0 0.5480 0 1 0 1 0.46 0 1 0 0.5230 0 1 0 -1 0.20 0 1 0 0.3010 0 1 0 -1 0.38 1 0 0 0.5350 0 1 0 1 0.50 0 1 0 0.5860 0 1 0 1 0.33 0 1 0 0.4250 0 1 0 -1 0.33 0 1 0 0.3930 0 1 0 1 0.26 0 1 0 0.4040 1 0 0 1 0.58 1 0 0 0.7070 1 0 0 1 0.43 0 0 1 0.4800 0 1 0 -1 0.46 1 0 0 0.6440 1 0 0 1 0.60 1 0 0 0.7170 1 0 0 -1 0.42 1 0 0 0.4890 0 1 0 -1 0.56 0 0 1 0.5640 0 0 1 -1 0.62 0 1 0 0.6630 0 0 1 -1 0.50 1 0 0 0.6480 0 1 0 1 0.47 0 0 1 0.5200 0 1 0 -1 0.67 0 1 0 0.8040 0 0 1 -1 0.40 0 0 1 0.5040 0 1 0 1 0.42 0 1 0 0.4840 0 1 0 1 0.64 1 0 0 0.7200 1 0 0 -1 0.47 1 0 0 0.5870 0 0 1 1 0.45 0 1 0 0.5280 0 1 0 -1 0.25 0 0 1 0.4090 1 0 0 1 0.38 1 0 0 0.4840 1 0 0 1 0.55 0 0 1 0.6000 0 1 0 -1 0.44 1 0 0 0.6060 0 1 0 1 0.33 1 0 0 0.4100 0 1 0 1 0.34 0 0 1 0.3900 0 1 0 1 0.27 0 1 0 0.3370 0 0 1 1 0.32 0 1 0 0.4070 0 1 0 1 0.42 0 0 1 0.4700 0 1 0 -1 0.24 0 0 1 0.4030 1 0 0 1 0.42 0 1 0 0.5030 0 1 0 1 0.25 0 0 1 0.2800 0 0 1 1 0.51 0 1 0 0.5800 0 1 0 -1 0.55 0 1 0 0.6350 0 0 1 1 0.44 1 0 0 0.4780 0 0 1 -1 0.18 1 0 0 0.3980 1 0 0 -1 0.67 0 1 0 0.7160 0 0 1 1 0.45 0 0 1 0.5000 0 1 0 1 0.48 1 0 0 0.5580 0 1 0 -1 0.25 0 1 0 0.3900 0 1 0 -1 0.67 1 0 0 0.7830 0 1 0 1 0.37 0 0 1 0.4200 0 1 0 -1 0.32 1 0 0 0.4270 0 1 0 1 0.48 1 0 0 0.5700 0 1 0 -1 0.66 0 0 1 0.7500 0 0 1 1 0.61 1 0 0 0.7000 1 0 0 -1 0.58 0 0 1 0.6890 0 1 0 1 0.19 1 0 0 0.2400 0 0 1 1 0.38 0 0 1 0.4300 0 1 0 -1 0.27 1 0 0 0.3640 0 1 0 1 0.42 1 0 0 0.4800 0 1 0 1 0.60 1 0 0 0.7130 1 0 0 -1 0.27 0 0 1 0.3480 1 0 0 1 0.29 0 1 0 0.3710 1 0 0 -1 0.43 1 0 0 0.5670 0 1 0 1 0.48 1 0 0 0.5670 0 1 0 1 0.27 0 0 1 0.2940 0 0 1 -1 0.44 1 0 0 0.5520 1 0 0 1 0.23 0 1 0 0.2630 0 0 1 -1 0.36 0 1 0 0.5300 0 0 1 1 0.64 0 0 1 0.7250 1 0 0 1 0.29 0 0 1 0.3000 0 0 1 -1 0.33 1 0 0 0.4930 0 1 0 -1 0.66 0 1 0 0.7500 0 0 1 -1 0.21 0 0 1 0.3430 1 0 0 1 0.27 1 0 0 0.3270 0 0 1 1 0.29 1 0 0 0.3180 0 0 1 -1 0.31 1 0 0 0.4860 0 1 0 1 0.36 0 0 1 0.4100 0 1 0 1 0.49 0 1 0 0.5570 0 1 0 -1 0.28 1 0 0 0.3840 1 0 0 -1 0.43 0 0 1 0.5660 0 1 0 -1 0.46 0 1 0 0.5880 0 1 0 1 0.57 1 0 0 0.6980 1 0 0 -1 0.52 0 0 1 0.5940 0 1 0 -1 0.31 0 0 1 0.4350 0 1 0 -1 0.55 1 0 0 0.6200 0 0 1 1 0.50 1 0 0 0.5640 0 1 0 1 0.48 0 1 0 0.5590 0 1 0 -1 0.22 0 0 1 0.3450 1 0 0 1 0.59 0 0 1 0.6670 1 0 0 1 0.34 1 0 0 0.4280 0 0 1 -1 0.64 1 0 0 0.7720 0 0 1 1 0.29 0 0 1 0.3350 0 0 1 -1 0.34 0 1 0 0.4320 0 1 0 -1 0.61 1 0 0 0.7500 0 0 1 1 0.64 0 0 1 0.7110 1 0 0 -1 0.29 1 0 0 0.4130 1 0 0 1 0.63 0 1 0 0.7060 1 0 0 -1 0.29 0 1 0 0.4000 1 0 0 -1 0.51 1 0 0 0.6270 0 1 0 -1 0.24 0 0 1 0.3770 1 0 0 1 0.48 0 1 0 0.5750 0 1 0 1 0.18 1 0 0 0.2740 1 0 0 1 0.18 1 0 0 0.2030 0 0 1 1 0.33 0 1 0 0.3820 0 0 1 -1 0.20 0 0 1 0.3480 1 0 0 1 0.29 0 0 1 0.3300 0 0 1 -1 0.44 0 0 1 0.6300 1 0 0 -1 0.65 0 0 1 0.8180 1 0 0 -1 0.56 1 0 0 0.6370 0 0 1 -1 0.52 0 0 1 0.5840 0 1 0 -1 0.29 0 1 0 0.4860 1 0 0 -1 0.47 0 1 0 0.5890 0 1 0 1 0.68 1 0 0 0.7260 0 0 1 1 0.31 0 0 1 0.3600 0 1 0 1 0.61 0 1 0 0.6250 0 0 1 1 0.19 0 1 0 0.2150 0 0 1 1 0.38 0 0 1 0.4300 0 1 0 -1 0.26 1 0 0 0.4230 1 0 0 1 0.61 0 1 0 0.6740 1 0 0 1 0.40 1 0 0 0.4650 0 1 0 -1 0.49 1 0 0 0.6520 0 1 0 1 0.56 1 0 0 0.6750 1 0 0 -1 0.48 0 1 0 0.6600 0 1 0 1 0.52 1 0 0 0.5630 0 0 1 -1 0.18 1 0 0 0.2980 1 0 0 -1 0.56 0 0 1 0.5930 0 0 1 -1 0.52 0 1 0 0.6440 0 1 0 -1 0.18 0 1 0 0.2860 0 1 0 -1 0.58 1 0 0 0.6620 0 0 1 -1 0.39 0 1 0 0.5510 0 1 0 -1 0.46 1 0 0 0.6290 0 1 0 -1 0.40 0 1 0 0.4620 0 1 0 -1 0.60 1 0 0 0.7270 0 0 1 1 0.36 0 1 0 0.4070 0 0 1 1 0.44 1 0 0 0.5230 0 1 0 1 0.28 1 0 0 0.3130 0 0 1 1 0.54 0 0 1 0.6260 1 0 0 -1 0.51 1 0 0 0.6120 0 1 0 -1 0.32 0 1 0 0.4610 0 1 0 1 0.55 1 0 0 0.6270 1 0 0 1 0.25 0 0 1 0.2620 0 0 1 1 0.33 0 0 1 0.3730 0 0 1 -1 0.29 0 1 0 0.4620 1 0 0 1 0.65 1 0 0 0.7270 1 0 0 -1 0.43 0 1 0 0.5140 0 1 0 -1 0.54 0 1 0 0.6480 0 0 1 1 0.61 0 1 0 0.7270 1 0 0 1 0.52 0 1 0 0.6360 1 0 0 1 0.3 0 1 0 0.3350 0 0 1 1 0.29 1 0 0 0.3140 0 0 1 -1 0.47 0 0 1 0.5940 0 1 0 1 0.39 0 1 0 0.4780 0 1 0 1 0.47 0 0 1 0.5200 0 1 0 -1 0.49 1 0 0 0.5860 0 1 0 -1 0.63 0 0 1 0.6740 0 0 1 -1 0.3 1 0 0 0.3920 1 0 0 -1 0.61 0 0 1 0.6960 0 0 1 -1 0.47 0 0 1 0.5870 0 1 0 1 0.3 0 0 1 0.3450 0 0 1 -1 0.51 0 0 1 0.5800 0 1 0 -1 0.24 1 0 0 0.3880 0 1 0 -1 0.49 1 0 0 0.6450 0 1 0 1 0.66 0 0 1 0.7450 1 0 0 -1 0.65 1 0 0 0.7690 1 0 0 -1 0.46 0 1 0 0.5800 1 0 0 -1 0.45 0 0 1 0.5180 0 1 0 -1 0.47 1 0 0 0.6360 1 0 0 -1 0.29 1 0 0 0.4480 1 0 0 -1 0.57 0 0 1 0.6930 0 0 1 -1 0.2 1 0 0 0.2870 0 0 1 -1 0.35 1 0 0 0.4340 0 1 0 -1 0.61 0 0 1 0.6700 0 0 1 -1 0.31 0 0 1 0.3730 0 1 0 1 0.18 1 0 0 0.2080 0 0 1 1 0.26 0 0 1 0.2920 0 0 1 -1 0.28 1 0 0 0.3640 0 0 1 -1 0.59 0 0 1 0.6940 0 0 1
.NET Test Automation Recipes
Software Testing
SciPy Programming Succinctly
Keras Succinctly
R Programming
Visual Studio Live
Microsoft MLADS Conference
DevIntersection Conference
Machine Learning Week
Ai4 Conference
G2E Conference
iSC West Conference
You must be logged in to post a comment.