Activation functions: sigmoid, tanh, ReLU and why nonlinearity matters
Activation functions insert nonlinearity into neural networks, enabling them to model complex relationships. The sigmoid outputs 0 to 1, tanh outputs -1 to 1, and ReLU returns 0 for negative inputs and the input for positive ones. ReLU is often preferred because it is simple and reduces the vanishing gradient problem.
Why nonlinearity matters
A neural network built only from linear operations, such as affine transformations, remains linear even when layers are stacked. Stanford's CS231n course notes explain that without a non-linearity, the weight matrices could be collapsed into a single matrix, making the network a linear function of the input.
Linear models also impose monotonicity: every feature increase must either always raise or always lower the output. Dive into Deep Learning gives the example that health risk rises for temperatures both above and below normal body temperature, which no linear model can capture, and argues that hidden layers with nonlinear activations overcome this limitation.
Activation functions apply a nonlinear transform to each neuron's output before it passes to the next layer, enabling the network to learn very complicated input-output relationships.
The three common activation functions
The sigmoid function, also called the logistic function, maps input x to 1/(1+e^{-x}) and produces outputs between 0 and 1. The tanh function, F(x)=tanh(x), outputs between -1 and 1. ReLU, defined as max(0,x), returns 0 for negative inputs and the input itself for non-negative inputs. According to Google's Machine Learning Crash Course, ReLU often works a little better than sigmoid or tanh because it is less susceptible to the vanishing gradient problem and is easier to compute.
Sigmoid and tanh are smooth and differentiable, while ReLU is piecewise linear and not differentiable at zero. These properties lead to different training behaviors, making each activation function suitable for specific roles in a network.
ReLU research and variants
PyTorch's ReLU documentation defines ReLU as the element-wise max(0, x) operation and notes that it can be performed in-place when desired. Link to PyTorch
The arXiv paper 'Delving Deep into Rectifiers' proposes a Parametric ReLU (PReLU) that generalizes the traditional rectified unit. The authors report that PReLU improves model fitting with nearly zero extra computational cost and achieves a 4.94% top-5 error on ImageNet, surpassing human-level performance at 5.1% and the ILSVRC winner's 6.66%. Link to arXiv
Choosing and implementing activation functions
Keras's activation functions documentation provides many built-in activation functions, including sigmoid, tanh, ReLU, ELU, GELU, and others. They can be passed as strings to layers, such as `Dense(64, activation='relu')`, or used as an `Activation` layer. Link to Keras
The choice of activation function can significantly affect network performance. While ReLU is a common default, other functions may be better for specific tasks such as classification or regression.
| Function | Output range | Formula | Characteristics |
|---|---|---|---|
| Sigmoid | (0,1) | 1/(1+e^{-x}) | S-shaped, smooth, outputs positive values |
| Tanh | (-1,1) | tanh(x) | S-shaped, zero-centered, outputs can be negative |
| ReLU | [0,∞) | max(0,x) | Piecewise linear, sparse outputs, not differentiable at zero |
Frequently asked questions
What is an activation function?
An activation function is a nonlinear transform applied to a neuron's output before it is passed to the next layer. It enables the network to model complex relationships.
Why do neural networks need nonlinearity?
Without nonlinearity, stacking linear layers results in a linear function, which cannot capture complex patterns such as non-monotonic relationships. Activation functions introduce nonlinearity.
What is the output range of tanh?
The tanh function maps any real input to a value between -1 and 1.
Why is ReLU often preferred over sigmoid and tanh?
ReLU is less susceptible to the vanishing gradient problem and is significantly easier to compute, as noted by Google's Machine Learning Crash Course.
What is PReLU?
Parametric ReLU (PReLU) generalizes the traditional rectified unit by allowing a learnable slope for negative inputs. It was introduced in an arXiv paper that achieved better ImageNet accuracy.