Hyperparameters: what they control and how grid and random search tune
Hyperparameters are external settings that control the training process of a machine learning model, such as learning rate and batch size. Grid search exhaustively evaluates all combinations of specified values, while random search samples from distributions to explore the parameter space more efficiently. Both methods aim to find the hyperparameter configuration that minimizes the loss function and improves model performance.
What hyperparameters control
Hyperparameters are configuration variables that data scientists set before training begins to manage the learning process of a machine learning model, as explained by IBM's hyperparameter tuning overview. They differ from model parameters, which are learned from data during training. Hyperparameters govern aspects such as model architecture, learning rate, and complexity.
Common hyperparameters include learning rate, which controls the step size during gradient descent; batch size, which determines how many training examples are processed before updating weights; and epochs, the number of times the entire dataset is passed through the model. According to Google's Machine Learning Crash Course, the learning rate influences how quickly the model converges, and batch size affects training speed and noise. Regularization hyperparameters, such as those controlling L1 or L2 penalties, help prevent overfitting by limiting model flexibility.
Proper hyperparameter tuning balances the bias-variance tradeoff, leading to accurate and consistent predictions. AWS's explanation of hyperparameter tuning states that hyperparameters directly control model structure, function, and performance, making their selection crucial for success.
Grid search
Grid search is an exhaustive method that evaluates all combinations of specified hyperparameter values. As described in the scikit-learn user guide on tuning, GridSearchCV generates candidates from a grid of parameter values and retains the best combination based on cross-validation scores. The user defines a discrete set of values for each hyperparameter, and the algorithm tries every possible combination.
This approach works well when the number of hyperparameters is small and the search space is limited. However, it becomes computationally expensive as the number of parameters grows, because the total number of combinations increases exponentially. AWS's overview notes that grid search is relatively tedious and computationally intensive, especially with large numbers of hyperparameters.
In practice, grid search is often used with cross-validation to evaluate each combination on different data splits. The best configuration is selected based on a performance metric such as accuracy or loss. It is a straightforward method that guarantees finding the optimal combination within the defined grid, but it may miss better values outside the specified ranges.
Random search
Random search samples hyperparameter values from specified distributions rather than evaluating all combinations. The scikit-learn documentation on randomized search explains that RandomizedSearchCV samples a given number of candidates from a parameter space with a specified distribution, allowing a fixed budget of trials. This method is more efficient than grid search when only a few hyperparameters significantly affect performance.
Random search supports both discrete and continuous hyperparameters. According to Microsoft Learn's guide on hyperparameter tuning, random sampling can use distributions such as uniform, normal, or log-uniform to define the search space. It also supports early termination of low-performing trials, which can save computational resources.
Because random search does not require evaluating all combinations, it can explore a wider range of values and is less affected by the curse of dimensionality. AWS's overview states that random search works well when a relatively small number of hyperparameters primarily determine the model outcome. It is often used as a first step to identify promising regions before applying more focused methods.
| Aspect | Grid search | Random search |
|---|---|---|
| Search method | Exhaustive evaluation of all combinations | Random sampling from distributions |
| Parameter space definition | Discrete list of values for each hyperparameter | Probability distributions or discrete choices |
| Computation cost | High, especially with many hyperparameters | Controlled by number of trials, independent of parameter count |
| Scalability | Poor; number of combinations grows exponentially | Good; adding parameters does not increase trials |
| Risk of missing optimum | Low within the defined grid, but may miss values outside | Higher, but can explore a broader range |
| Typical use case | Small number of hyperparameters with few values | Large search space or many hyperparameters |
Frequently asked questions
What is a hyperparameter?
A hyperparameter is a configuration variable set before training begins to control the learning process of a machine learning model, such as learning rate, batch size, or the number of hidden layers.
How do hyperparameters differ from model parameters?
Hyperparameters are set manually by the data scientist before training, while model parameters (such as weights and biases) are learned automatically from the data during training.
What is grid search?
Grid search is an exhaustive tuning method that evaluates all possible combinations of specified hyperparameter values to find the best configuration, using cross-validation to assess performance.
What is random search?
Random search is a tuning method that samples hyperparameter values from defined distributions, running a fixed number of trials to explore the search space more efficiently than grid search.
Which method is more efficient for large search spaces?
Random search is generally more efficient for large search spaces because it can explore a wider range with a fixed budget, whereas grid search becomes computationally prohibitive as the number of hyperparameters grows.