Training, validation and test sets: why data is split first
Machine learning models are evaluated on data they have not seen during training. Splitting the original dataset into training, validation, and test sets prevents overfitting and gives a realistic estimate of performance. The split must happen before any preprocessing or model selection to avoid data leakage.
The three-way split
In supervised machine learning, best practice is to split the available data into three independent sets: a training set, a validation set, and a test set, as explained by the MLU Explain guide. The training set is used to teach the model underlying patterns. The validation set provides an unbiased way to compare different model types and hyperparameter choices. The test set approximates the model's accuracy on unseen data.
The Google for Developers course on dividing datasets recommends this three-way split instead of a simple two-way split. A validation set performs initial testing while the model is being trained. After repeated use of the validation set suggests the model is making good predictions, the test set is used for a final check.
The test set should only be used once, at the end of the process. Amazon MLU Explain states that peeking at test set performance before selecting a model is a form of overfitting and leads to unreliable expectations in production.
Why the split must happen first
The order of operations matters. The scikit-learn documentation on common pitfalls advises always splitting the data into train and test subsets first, particularly before any preprocessing steps. If preprocessing like scaling or feature selection is applied to the whole dataset before splitting, information from the test set can influence the model, a problem called data leakage.
Data leakage means that details which would be unknown at prediction time are used during model building. This makes performance estimates look better than they are and hurts results on new data, as noted in the scikit-learn pitfalls page. The general rule is to never call fit on the test data.
The same page gives a concrete example: if a normalization step divides by an average value, that average must be computed from the training subset only. Including the test subset in the average calculation leaks information. Using a pipeline in scikit-learn can help ensure that the correct transformation is applied to each subset.
The role of the validation set
The validation set exists to prevent overfitting to the test set. The scikit-learn user guide on cross-validation explains that when hyperparameters are tuned repeatedly on the same test set, the model can start to fit that test set, so evaluation metrics no longer reflect true generalization. A separate validation set allows tuning without touching the test set.
The Stanford CS231n notes on image classification describe using a validation set for hyperparameter tuning. Instead of testing on the test set during development, one can split the training data into a training set and a validation set, train on the training set, and evaluate on the validation set to choose hyperparameters. The test set is reserved for the final evaluation.
Both the validation and test sets can wear out with repeated use. The Google for Developers dividing datasets page notes that the more often the same data is used to make decisions about hyperparameters or model improvements, the less confidence there is that the model will perform well on new data. Collecting fresh data to refresh these sets is recommended.
Cross-validation as an alternative
When data is scarce, partitioning into three sets can drastically reduce the number of samples available for learning. The scikit-learn cross-validation guide presents cross-validation as a solution. In k-fold cross-validation, the training set is split into k smaller sets. For each fold, a model is trained on k-1 folds and validated on the remaining fold. The reported performance is the average across all folds.
Cross-validation does not waste as much data as a fixed validation set, which is a major advantage when the number of samples is very small. However, a test set should still be held out for final evaluation. The scikit-learn guide shows an example with the iris dataset where a linear support vector machine achieves a mean accuracy of 0.98 with a standard deviation of 0.02 over five folds.
The IBM article on training data emphasizes that the quality and quantity of training data are both important. Poor quality data can introduce noise and bias, preventing accurate predictions. High-quality training data allows models to produce more reliable results across use cases.
| Pitfall | Consequence | Recommended practice |
|---|---|---|
| Splitting after preprocessing | Test data influences preprocessing, causing data leakage | Split first, then fit preprocessing on training data only |
| Using test set for hyperparameter tuning | Model overfits to test set, performance estimate is optimistic | Use a validation set or cross-validation for tuning |
| Applying different transformations to train and test | Feature space changes, model performs poorly on test data | Apply the same transformation to all subsets, e.g., using a pipeline |
| Reusing the same test set many times | Test set wears out, confidence in predictions decreases | Collect fresh data to refresh test and validation sets |
| Including duplicates of training examples in test set | Test loss is artificially low | Remove duplicate examples from the test set |
Frequently asked questions
Why not train and test on the same data?
Testing on the same data used for training gives an overly optimistic performance estimate. A model that simply repeats the labels of samples it has seen would score perfectly but fail on new data, a situation called overfitting, as explained in the scikit-learn cross-validation guide.
What is data leakage and how can it be avoided?
Data leakage occurs when information not available at prediction time is used to build the model, such as including test data in preprocessing calculations. It leads to optimistic performance estimates. Avoid it by splitting data first and never calling fit on test data, as recommended by scikit-learn's common pitfalls page.
When should the test set be used?
The test set should be used only once, as the final evaluation after model selection and hyperparameter tuning are complete. Peeking at test performance earlier is a form of overfitting, according to Amazon MLU Explain.
What is the advantage of cross-validation over a fixed validation set?
Cross-validation uses the data more efficiently by training and validating on multiple folds, which is helpful when the number of samples is small. It reduces the dependence on a particular random split, as described in the scikit-learn cross-validation guide.
Can validation and test sets wear out?
Yes. Repeated use of the same validation or test set to make decisions reduces confidence that the model will generalize to new data. Google for Developers recommends collecting more data to refresh these sets.