Hyperbolic Tangent

The tanh analyzes diversity in the input space similar to SVMs. This scales well to high dimensional spaces where tanh does a multidimensional diversity analysis.

Here we use the Hessian Normal Form to rewrite the sum into a dot product. We can calculate $b$ to be $\frac{w_0}{w_i}$.
Then we can use the normal dot product rule to rewrite it into the product of the two magnitudes multiplied by the angle between them. Here we can first see that the result of the neuron depends both on the distance to the bias and the angle between the weight and the input. Geometrically we can view it like this:

Multiple dimensions:

Approximate Normal Distribution

when $u$ is not zero the network can also model with RBFs.

First Derivative

Simplifications

The following simplification is an easy to compute approximation of the tanh function. But it is non-differentiable in two points and we multiply the error flow with zero when leaving the middle range.

In Backpropagation we have

which would be zero.

The following two approximations are quite similar to tanh, they all go through the origin and are easier to compute.
But you can see that the curvature of the functions is slower growing, thus the model will try to learn bigger weights to reach the entire -1, +1 interval.
In generalization this will lead to an unstable model as small changes in the input data will be amplified by the large weights.