Multilayer perceptrons: how hidden layers learn nonlinear patterns
Multilayer perceptrons use hidden layers to capture nonlinear relationships in data. Activation functions prevent the network from collapsing into a linear model. Training via backpropagation automatically learns feature crosses, enabling the model to represent complex patterns.
Why linear models are not enough
A linear model computes an output as a weighted sum of inputs plus a bias. The Dive into Deep Learning textbook notes that linearity implies monotonicity: increasing any feature always moves the output in one direction, which fails for cases like body temperature and health risk, where both low and high values are dangerous. For image classification, a linear model cannot account for the context of pixels, since inverting an image preserves the category but a linear function would treat it differently.
Feature crosses can let a linear model handle some nonlinear relationships, but choosing the right crosses requires manual experimentation. The Google Machine Learning Crash Course explains that neural networks are designed to automatically find the optimal feature crosses during training, removing the burden of hand-crafting them.
Deep neural networks address this by learning a representation through hidden layers and then applying a linear predictor on top of that representation, as described in the same textbook. This allows the model to capture complex dependencies from raw features.
Architecture: hidden layers and neurons
A hidden layer sits between the input and output layers. The Google page on nodes and hidden layers defines hidden layers and neurons: each neuron takes the sum of its inputs multiplied by unique weights, adds a bias, and passes the result to the next layer.
That page also presents a concrete example: a network with three inputs, one hidden layer of four neurons, and one output has 21 parameters. The hidden layer uses 12 weights and 4 biases, while the output layer has 4 weights and 1 bias.
A fully connected layer is an affine transformation. The PyTorch documentation shows that torch.nn.Linear applies y = xA^T + b, where A and b are learnable parameters, with the weight matrix initialized from a uniform distribution influenced by the input feature count.
Activation functions introduce nonlinearity
If a network only contains affine layers, stacking them still produces a linear transformation. The Stanford CS231n notes state that without a non-linearity, the two matrices can be collapsed into one, so the predicted class scores remain a linear function of the input.
Common activation functions include ReLU (max(0, x)) and sigmoid, as described in the same notes. A two-layer network might compute s = W2 max(0, W1 x), where the activation is applied elementwise.
The non-linearity gives the model its ability to approximate curved decision boundaries and capture interactions between features, which is why hidden layers are effective for nonlinear problems.
Training and automatic feature crosses
The Google Machine Learning Crash Course explains that neural networks are trained using the backpropagation algorithm, which optimizes predictions and minimizes loss by adjusting weights and biases.
During training, the network automatically learns the optimal feature crosses to perform on the input data, as the course notes. This is why multilayer perceptrons can discover nonlinear patterns without manual feature engineering.
The combination of hidden layers, activation functions, and backpropagation enables the model to jointly learn a representation and a linear predictor, as highlighted in the deep learning textbook.
| Model | Hidden neurons | Weights | Biases | Total parameters |
|---|---|---|---|---|
| Linear model (no hidden layer) | 0 | 3 | 1 | 4 |
| Multilayer perceptron with one hidden layer | 4 | 16 | 5 | 21 |
Frequently asked questions
What is a hidden layer?
A hidden layer is a set of neurons between the input and output layers. It recombines input signals using learnable weights and biases, then passes the result through an activation function.
Why are activation functions necessary in a multilayer perceptron?
Without activation functions, stacked affine layers still produce a linear model, as the composition of linear transformations is linear. Activations like ReLU or sigmoid introduce nonlinearity, allowing the network to model complex patterns.
How many parameters does the Google example network have?
A network with three inputs, one hidden layer of four neurons, and one output has 21 parameters: 16 weights and 5 biases. The hidden layer contributes 12 weights and 4 biases, and the output layer contributes 4 weights and 1 bias.
How does a multilayer perceptron learn nonlinear patterns?
Training uses backpropagation to minimize loss. The network automatically adjusts weights and biases, which lets it discover relevant feature interactions without manual feature crossing.
What does torch.nn.Linear do?
It applies an affine transformation y = xA^T + b, where A is a learnable weight matrix and b is a learnable bias vector, as documented by PyTorch.