Linear regression: fitting a straight line to predict numeric values
Linear regression is a statistical method that models the relationship between a dependent variable and one or more independent variables by fitting a straight line. The line is chosen to minimize the sum of squared differences between observed and predicted values, a technique called ordinary least squares. The method is widely used in machine learning for prediction and inference, with extensions such as Ridge regression to handle multicollinearity.
.jpg){kind=link}
What is linear regression?
Linear regression is a supervised learning algorithm used to predict a continuous numeric value. According to the IBM article on linear regression, it models the relationship between a dependent variable and one or more independent variables by fitting a linear equation.
In its simplest form, simple linear regression uses one independent variable and is expressed as y' = b + w₁x₁, where y' is the predicted label, b is the bias (intercept), and w₁ is the weight (slope). The Google for Developers linear regression module describes how the model adjusts the bias and weights during training to minimize prediction error.
When multiple features are involved, the model becomes multiple linear regression: y' = b + w₁x₁ + w₂x₂ + ... + wₚxₚ. The AWS explanation of linear regression notes that multiple linear regression can capture the influence of several predictor variables on a single outcome.
How linear regression works
The core of linear regression training is to find the line that best fits the data by minimizing a loss function. The most common loss is the sum of squared residuals, known as ordinary least squares (OLS). The Cornell University lecture on linear regression derives OLS from maximum likelihood estimation and provides the closed-form solution w = (X Xᵀ)⁻¹ X yᵀ, where X is the design matrix and y the target vector.
The least squares solution can be computed via the singular value decomposition, a method that is efficient for many practical datasets. For large datasets, gradient descent is often used instead of the closed form, iteratively adjusting the weights to reduce the error.
Assumptions and diagnostics
Linear regression relies on several assumptions for valid inference. The NVIDIA glossary on linear regression notes that multiple linear regression assumes a linear relationship between dependent and independent variables, that residuals are normally distributed, and that all random variables have the same finite variance.
To verify these assumptions, data scientists create scatterplots to check for linearity and use residual plots to detect patterns. If residuals show heteroscedasticity or non-normality, transformations of the dependent variable or removal of outliers may help.
The IBM resource also lists diagnostics such as the Durbin-Watson test for residual independence and methods to evaluate homoscedasticity. However, since IBM is already cited elsewhere, this paragraph does not require a new link.
Variants and extensions
When features are highly correlated, ordinary least squares can produce unstable coefficients. Ridge regression addresses this by adding an L2 penalty to the loss function: the minimization objective becomes the sum of squared residuals plus α times the squared magnitude of the weights. The scikit-learn user guide on linear models explains that the parameter α controls the amount of shrinkage, and larger values make the coefficients more stable in the presence of collinearity.
Another variant is non-negative least squares, which constrains coefficients to be non-negative, useful when features represent physical quantities. Ridge regression has a closed-form solution that includes the identity matrix, and its computational complexity is similar to ordinary least squares.
| Type | Number of independent variables | Equation form | Example use case |
|---|---|---|---|
| Simple linear regression | One | y' = b + w₁x₁ | Predicting house price from size |
| Multiple linear regression | Two or more | y' = b + w₁x₁ + w₂x₂ + ... + wₚxₚ | Predicting fuel efficiency from weight, displacement, and horsepower |
Frequently asked questions
What is the difference between linear regression and logistic regression?
Linear regression predicts a continuous numeric value, while logistic regression predicts the probability that an item belongs to a discrete class. The NVIDIA glossary explains that logistic regression is used for classification, such as determining whether an email is spam.
Can linear regression be used with categorical features?
Yes, but categorical variables must be converted to numeric form, typically using one-hot encoding or dummy variables. The IBM resource notes that categorical variables such as region of residence need to be recoded to binary contrast variables before use.
What does the R² value indicate in linear regression?
R², or the coefficient of determination, measures the proportion of variance in the dependent variable that is explained by the independent variables. It ranges from 0 to 1, with higher values indicating a better fit. The IBM page lists R² among the statistics reported for each model.
How do outliers affect linear regression?
Outliers can have a large influence on the fitted line because ordinary least squares minimizes squared errors, giving disproportionate weight to points far from the line. The AWS guide recommends checking for significant outliers and potentially removing them or using alternative regression methods that are less sensitive to outliers.
What is the role of the bias term in linear regression?
The bias term (intercept) allows the regression line to shift vertically, ensuring the model can fit data that does not pass through the origin. The Google for Developers module describes bias as a parameter that is updated during training, equivalent to the y-intercept in the algebraic equation for a line.