Loss functions: how a model measures its errors during training
A loss function measures how far a model's predictions are from the actual labels. Training minimizes this value by adjusting model parameters. Different tasks use different loss functions, such as squared error for regression and log loss for classification.
.png){kind=link}
What a loss function measures
A loss function measures how wrong a model's predictions are by calculating the deviation from the ground truth. IBM's explanation of loss functions describes it as a measure of model performance that is minimized during training. When predictions are accurate, the loss is small; when they are not, the loss is large.
Loss functions are used in supervised learning, where labeled examples provide the correct answers. The training process adjusts parameters to reduce the loss below a predetermined threshold. This makes the loss function the mathematical objective for learning.
Regression loss functions
For regression, loss measures the distance between the predicted value and the actual value, ignoring the sign. Two common ways to remove the sign are taking the absolute value or squaring the difference. According to Google for Developers' linear regression module, the mean absolute error (MAE) averages the absolute differences, while the mean squared error (MSE) averages the squared differences. Squaring makes MSE more sensitive to outliers than MAE.
Cornell University's lecture notes on empirical risk minimization identify several regression losses. Squared loss, also called ordinary least squares, estimates the mean label and is differentiable everywhere. Absolute loss estimates the median and is less sensitive to noise but not differentiable at zero. Huber loss combines both, acting like squared loss for small errors and absolute loss for large errors.
Classification loss functions
In classification, a loss function quantifies the agreement between predicted scores and ground truth labels. Stanford's CS231n notes describe the loss function as part of an optimization problem where the goal is to set parameters so that correct classes receive higher scores than incorrect ones. The multiclass support vector machine uses hinge loss, which penalizes predictions that are close to or on the wrong side of the decision boundary.
For logistic regression, the loss function is log loss. As Google for Developers' logistic regression course explains, log loss is used because the sigmoid function has a non-constant rate of change, so squared loss would require excessive precision near the extremes. Log loss is defined as the negative average of the logarithms of the predicted probabilities. Regularization is also critical in logistic regression to prevent overfitting.
Optimization and regularization
To minimize loss, training uses optimization algorithms such as gradient descent. The loss function must be differentiable so that its gradient can be calculated. For neural networks, backpropagation computes the gradient by passing the loss backward through the network. Parameters are then updated in the direction that reduces the loss.
Regularization adds a penalty term to the loss to discourage complex models. This helps models generalize beyond the training data. Common forms include L2 regularization, which penalizes the sum of squared weights, and early stopping, which halts training while loss is still decreasing to avoid overfitting.
| Loss function | Task | Notable property |
|---|---|---|
| Mean Absolute Error (MAE) | Regression | Less sensitive to outliers |
| Mean Squared Error (MSE) | Regression | Penalizes outliers more heavily |
| Huber Loss | Regression | Combines squared and absolute loss |
| Hinge Loss | Classification | Used in support vector machines |
| Log Loss | Classification | Used in logistic regression |
| Exponential Loss | Classification | Used in AdaBoost, aggressive on mispredictions |
| Zero-One Loss | Classification | Actual classification error, not differentiable |
Frequently asked questions
What is the difference between mean absolute error and mean squared error?
MAE averages the absolute differences, while MSE averages the squared differences. MSE penalizes large errors more heavily and can be more sensitive to outliers.
Why is log loss used for logistic regression?
Because the sigmoid function's rate of change is not constant, and log loss measures the logarithm of the magnitude of the change, avoiding the need for extreme precision near the asymptotes.
What does it mean for a loss function to be differentiable?
It means the function has a derivative at every point, which is required for gradient-based optimization.
How does regularization help avoid overfitting?
Regularization adds a penalty to the loss function based on model complexity, discouraging large weights and encouraging simpler models that generalize better.
Can a loss function be used as an evaluation metric?
Loss functions are primarily for training, but some like MAE and RMSE can also be used to report model error because they are interpretable on the same scale as predictions.