I’ve been exploring the idea of implementing neural networks using raw JavaScript. None of the individual parts of a neural network (weights and biases, initialization, input-output, etc.) are hugely complicated by themselves, but there are a lot of parts.

One task to deal with when implementing a neural network from scratch is activation functions. Simple one-hidden-layer neural nets typically use tanh or logistic sigmoid activation on the hidden nodes and, for NN classifiers, softmax activation on the output nodes.

Note that for deep neural networks with several hidden layers, ReLU (rectified linear unit) activation is often used, but implementing a deep NN from scratch isn’t a common scenario because of the availability of deep neural libraries such as TensorFlow/Keras and PyTorch.

I coded up demos of tanh, logistic sigmoid, and softmax activation using JavaScript. There were no surprises and I didn’t run into any big problems.

For my demo softmax function, I used a naive approach. The idea of softmax is to scale two or more arbitrary numeric values so that they sum to 1.0 and can be interpreted as probabilities. For example, softmax([3.0, 5.0, 2.0]) = [0.12, 0.84, 0.14].

For the naive approach, you compute the sum of the exp() of each value, then divide the exp() of each value by the sum:

exp(3.0) =  20.09
exp(5.0) = 148.41
exp(2.0) =   7.39
-----------------
sum =      175.89

softmax = [20.09/175.89, 148.41/175.89, 7.39/175.89]
        = [0.12, 0.84, 0.14]

Notice that if a source value is even moderately sized, the exp() of it could be extremely large, which could cause numeric overflow or underflow that you have to watch for. For example, exp(30.0) is about 10686000000000.0 — yikes. There is a technique called the max trick you can use to reduce the chance of this happening See

The Max Trick when Computing Softmax

for details.

Large values and overflow may be trouble for computer systems, but aren’t a problem in Hollywood. From left: Petite actress Audrey Hepburn had size 11 feet. Uma Thurman has very large hands. Farrah Fawcett was iconic for her big hair in the 1980s. Famous photo where actress Sophia Loren observes Jayne Mansfield overflow.