I wrote an article titled Customize Neural Networks with Alternative Activation Functions” in the June 2015 issue of Visual Studio Magazine. See https://visualstudiomagazine.com/articles/2015/06/01/alternative-activation-functions.aspx.

You can think of a neural network as a complicated math function that is used to make predictions. For example, you might want to predict the political leaning (conservative, moderate, liberal) of a person based on predictor variables like age, income, sex (-1 = male, +1 = female), and so on.

Inside a neural network, something called a hidden node activation function is used. By far the two most common activation functions are the hyperbolic tangent (abbreviated tanh) and the logistic-sigmoid (log-sigmoid). The tanh function accepts any numeric value and returns a value between -1.0 and +1.0. The closely related log-sigmoid function accepts any numeric value and returns a value between 0.0 and +1.0.

In my article I explain that there are dozens of alternative activation functions possible, and explain that tanh and log-sigmoid are by far the most common because they are, by an algebra coincidence, slightly easier to work with than alternatives.

In the article I show how to use the arctangent (arctan) function for hidden node activation. The arctan accepts any numeric value and returns a value between approximately -1.6 and +1.6. The arctan example provides you with enough information to experiment with neural networks using any alternative activation function.