Cross-validation: estimating performance with repeated data splits
Cross-validation is a technique that repeatedly splits the training data into complementary subsets, training on one part and evaluating on the other. The most common form, k-fold cross-validation, averages the performance across k rounds to provide a reliable estimate of model generalization. It helps prevent overfitting and is essential for hyperparameter tuning.
What is cross-validation?
When training a machine learning model, evaluating it on the same data used for training can lead to overfitting. The scikit-learn user guide explains that this is a methodological mistake because a model that simply repeats the labels of samples it has seen would score perfectly but fail on unseen data.
To avoid overfitting, a portion of the data is often held out as a test set. However, a single validation set for hyperparameter tuning is problematic, so cross-validation uses multiple splits.
How k-fold cross-validation works
In k-fold cross-validation, the training set is divided into k equal-sized folds. The Cornell lecture notes describe the procedure: the data is split into k sections, and then k separate runs are performed. In each run, the model is fitted on k-1 sections and tested on the remaining section, so that every section serves once as a validation set, and the validation errors are averaged.
This yields an estimate of validation error with its standard deviation. For small datasets, leave-one-out cross-validation (k equals the number of samples) is an option.
Why not a single validation set?
A single validation set is just one random split, and the results can depend heavily on that split. The CS231n course notes point out that using a single validation set for hyperparameter tuning can leak information about the test set into the model, and the validation set may not be representative.
Cross-validation mitigates this by averaging over multiple splits, providing a more reliable estimate of generalization performance.
Cross-validation for hyperparameter tuning
Cross-validation is integral to hyperparameter optimization. The scikit-learn grid search documentation describes GridSearchCV, which exhaustively evaluates all parameter combinations using cross-validation, and RandomizedSearchCV, which samples a given number of candidates from a parameter space.
These tools use cross-validation to select the best parameters without overfitting to a specific validation set.
Avoiding data leakage
A common pitfall is data leakage, where information from the test set influences the model. The scikit-learn common pitfalls guide recommends splitting data into train and test subsets first, and never calling fit on the test data.
To avoid leakage in cross-validation, preprocessing steps should be learned only from training folds within the cross-validation loop, and using a Pipeline object ensures consistent application.
| Strategy | Description | Typical use case |
|---|---|---|
| k-fold | Data is split into k equal-sized folds; model trained on k-1 folds, tested on the remaining fold, repeated k times. | General purpose; default choice for many tasks. |
| Stratified k-fold | Same as k-fold but each fold preserves the class distribution of the whole dataset. | Classification with imbalanced classes. |
| Leave-one-out (LOOCV) | k equals the number of samples; each sample is used once as a test set. | Very small datasets where every sample matters. |
| Shuffle split | Generates independent train/test splits by randomly sampling without replacement. | When repeated random splits are desired, e.g., for large datasets. |
Frequently asked questions
What is the difference between a validation set and cross-validation?
A validation set is a single held-out portion of the training data used to tune hyperparameters. Cross-validation uses multiple splits, averaging results to reduce the variance of the performance estimate and make better use of limited data.
Why is k-fold cross-validation preferred over a single train-test split?
A single split can be sensitive to the random partition, especially with small datasets. K-fold cross-validation averages over k different splits, providing a more reliable estimate of model generalization.
What is leave-one-out cross-validation (LOOCV)?
LOOCV is a special case of k-fold where k equals the number of samples. Each sample is used once as a test set while the rest form the training set. It is computationally expensive but useful for very small datasets.
How is the number of folds k chosen?
Common choices are 5 or 10 folds. A larger k reduces bias but increases variance and computational cost. For very large datasets, even 3 folds can suffice. The choice depends on the dataset size and the trade-off between bias and variance.
Does cross-validation prevent overfitting?
Cross-validation helps detect overfitting by evaluating the model on multiple unseen subsets. It does not prevent overfitting during training, but it provides a more honest estimate of performance, guiding the selection of less complex models or better hyperparameters.