Artificial neurons: weights, biases and the weighted sum
An artificial neuron multiplies each input by a learned weight, sums the products, adds a bias, and then passes the result through an activation function. The weights and bias are trainable parameters that determine the neuron's behavior, while the activation function introduces necessary nonlinearity. This article explains the components with references from Stanford, Google, IBM, NVIDIA, and PyTorch.
What is an artificial neuron?
An artificial neuron is the basic computational unit of a neural network. It is loosely inspired by biological neurons, as described in the Stanford University CS231n course notes, but the engineering model is far simpler: each neuron receives multiple inputs, weighs them, and produces a single output. The biological analogy helps explain the structure, but modern usage is purely mathematical.
In this model, each input is multiplied by a corresponding weight, and the products are summed together with a bias. The resulting value is then passed through an activation function that determines the neuron's firing rate or output. This process mirrors the biological idea of dendrites collecting signals and an axon sending a spike when a threshold is reached.
Weights and the weighted sum
Weights represent the strength of connections between units in a network. According to the Stanford Encyclopedia of Philosophy's entry on connectionism, weights can be positive or negative, where a negative weight models inhibition. These weights are learnable parameters that are adjusted during training to map inputs to desired outputs.
The weighted sum is the core linear computation of a neuron. PyTorch's Linear module documentation formalizes this as an affine transformation: y = xA^T + b, where A holds the weights and b is the bias. This operation combines all inputs and the bias into a single scalar value before any nonlinear transformation is applied.
The role of bias
The bias is an additive constant that shifts the weighted sum. As explained in IBM's overview of neural networks, biases shift the decision threshold, enabling a neuron to fire even when inputs are weak. Without a bias, a neuron with zero inputs would always output zero, limiting its expressive power.
In a layer with multiple neurons, each neuron has its own bias term. The bias is trained alongside the weights, so it becomes an adjustable part of the model. Together, weights and biases form the parameters that define how a neuron transforms its inputs.
Activation functions and nonlinearity
An activation function applies a nonlinear transformation to the weighted sum. NVIDIA's developer article on artificial neural networks states that activation functions such as logistic sigmoid, tanh, and ReLU are essential for creating complex features. They introduce nonlinearity, which allows a network to represent relationships beyond simple linear mappings.
Without activation functions, a stack of linear layers collapses into a single linear transformation, because a chain of matrix multiplications is still a matrix multiplication. Thus, nonlinear activation functions are what give deep networks their power to approximate a wide range of functions. The choice of activation function affects how the network learns and performs.
Counting parameters in a hidden layer
In a fully connected layer, each neuron computes a weighted sum of all inputs plus a bias. Therefore, the number of parameters in a layer equals (number of inputs + 1) times the number of neurons. Google's nodes and hidden layers page provides a concrete example: a network with 3 inputs, a hidden layer of 4 neurons, and 1 output has 21 parameters total.
For the hidden layer, there are 4 neurons, each with 3 weights (12 weights) and one bias per neuron (4 biases), totaling 16 parameters. The output layer adds 4 weights and 1 bias, making 5 more parameters. The sum is 21, which matches the example in Google's exercise. Counting parameters is a straightforward way to gauge the size and complexity of a neural network.
| Layer | Weights | Biases | Parameters |
|---|---|---|---|
| Hidden layer | 12 | 4 | 16 |
| Output layer | 4 | 1 | 5 |
| Total | 16 | 5 | 21 |
Frequently asked questions
What is a weighted sum in a neuron?
The weighted sum is the sum of each input multiplied by its weight, plus the bias. It is the linear combination that the neuron computes before applying an activation function.
Why are biases needed?
Biases shift the decision threshold of a neuron, allowing it to activate even when the inputs are zero or weak. This adds flexibility to the model.
What is an activation function?
An activation function is a nonlinear transformation applied to the weighted sum. It enables neural networks to learn complex, nonlinear relationships between inputs and outputs.
How are weights and biases learned?
During training, the network uses backpropagation to adjust weights and biases so that the predicted output moves closer to the target value, minimizing a loss function.
How many parameters does a layer have?
A fully connected layer with n inputs and m neurons has (n + 1) * m parameters, comprising n weights per neuron plus one bias per neuron.