Logistic regression: predicting class probabilities with the sigmoid
Logistic regression is a supervised classification algorithm that predicts the probability of a categorical outcome. It uses the sigmoid function to transform a linear combination of features into a value between 0 and 1, which can be used directly or mapped to a binary label. Training typically maximizes the likelihood of the observed data using gradient descent, and regularization helps control overfitting.
Classification with probabilities
Logistic regression is a supervised learning method for classification that predicts a discrete outcome, such as whether an email is spam or not. Unlike linear regression, which predicts continuous numeric values, logistic regression produces a probability in the range from 0 to 1, based on the model's inputs. IBM's logistic regression explainer notes that the model fits an S-shaped curve to map inputs to probabilities rather than a straight line.
The method is widely used across industries. For example, AWS's logistic regression overview lists applications in manufacturing for predicting part failure, in healthcare for estimating disease likelihood, and in finance for fraud detection and risk assessment. The same page highlights the model's simplicity and low computational cost compared to other techniques.
The sigmoid function and log-odds
The sigmoid function, also called the logistic function, is central to logistic regression. Its formula is f(x) = 1 divided by (1 + e to the power of -x), and its output always lies strictly between 0 and 1, regardless of the input. Google's sigmoid function page explains that as the input grows, the output approaches 1 but never exactly reaches it, and as the input decreases, the output approaches 0.
In a logistic regression model, the linear equation z = b + w1x1 + ... + wNxN is computed first, then passed to the sigmoid function to obtain a probability. The value z is known as the log-odds because solving the sigmoid equation for z yields the natural logarithm of the ratio between the probability of the positive outcome and that of the negative outcome.
Training via maximum likelihood and gradient descent
Logistic regression parameters are estimated by maximizing the conditional likelihood of the observed labels given the features, an approach called maximum likelihood estimation (MLE). Cornell's lecture on logistic regression describes logistic regression as a discriminative model that directly models the probability of the label given the input, in contrast to generative models such as Naive Bayes.
Because the MLE objective has no closed-form solution, gradient descent is used to minimize the negative log likelihood. The Cornell lecture also notes that adding a Gaussian prior on the weights leads to a maximum a posteriori (MAP) estimate, which introduces a regularization penalty that can help prevent overfitting.
Regularization and implementation in practice
Regularization is essential when training logistic regression models, especially with many features. scikit-learn's user guide on linear models indicates that the module supports coefficients stored in coef_ and intercept_ attributes, and it points to a dedicated logistic regression section for classification. The guide also mentions that logistic regression is part of the linear models family.
Logistic regression uses log loss as its loss function rather than squared loss. Google's logistic regression module emphasizes this difference and outlines use cases such as spam detection and weather prediction. The module also stresses the importance of regularization for model generalization.
| Aspect | Linear Regression | Logistic Regression |
|---|---|---|
| Output type | Continuous numeric value | Probability between 0 and 1 |
| Prediction range | Unbounded | Bounded to (0,1) |
| Typical equation | y = β0 + β1x | p = 1 / (1 + e^(-z)), z = β0 + β1x |
| Loss function | Squared loss | Log loss |
Frequently asked questions
How does logistic regression differ from linear regression?
Linear regression predicts continuous numeric values, while logistic regression predicts the probability of a categorical outcome, always between 0 and 1.
What does the sigmoid function do?
It maps any real number to a value between 0 and 1, following the formula f(x)=1/(1+e^-x), ensuring the output is a valid probability.
What are log-odds in logistic regression?
The linear combination z = b + w1x1 + ... is called the log-odds, which is the natural logarithm of the ratio of the probability of the positive outcome to that of the negative outcome.
Why is regularization used in logistic regression?
Regularization adds a penalty to the objective function to shrink coefficients, reducing overfitting, especially when there are many features.