ML Hyperparameter Tuning Strategies

Hyperparameter tuning is where most modeling time is spent after the initial pipeline is built. The default values usually work, but the right values can lift accuracy noticeably. This post compares the main strategies, with notes on when each one earns its compute.

What hyperparameter tuning really is

Hyperparameters are settings that you fix before training, like learning rate, tree depth, regularization strength, or batch size. They are different from model parameters, which the optimizer learns. Tuning is the process of searching over hyperparameter values and picking the combination that scores best on a validation set.

The search has three components: the space (which hyperparameters and what ranges), the strategy (how to pick the next combination), and the budget (how many runs you can afford).

Mental model

Picture a knob panel. Each knob is a hyperparameter, each setting is a possible value. You want to find the combination of knobs that gives the best score, but every evaluation costs real training time. The strategy is how you decide which knob settings to try next.

define space
 |
 v
suggest config -> train -> evaluate -> log
 ^                                    |
 |____________________________________|
      until budget exhausted
                 |
                 v
          best config

Hands-on example

A typical setup with scikit-learn’s random search.

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint

space = {
    "n_estimators": randint(50, 500),
    "max_depth": randint(2, 8),
    "learning_rate": loguniform(1e-3, 0.3),
    "subsample": [0.6, 0.8, 1.0],
}

search = RandomizedSearchCV(
    GradientBoostingClassifier(),
    space, n_iter=30, cv=5, n_jobs=-1, random_state=0,
)
search.fit(X, y)
print(search.best_params_, search.best_score_)

For larger workloads, libraries like Optuna, Ray Tune, or Hyperopt add Bayesian optimization and early stopping. The interface looks similar: define a search space and a budget, and the library proposes configurations.

Trade-offs

Grid search is exhaustive over a discrete grid. It is simple, but the cost explodes with each new hyperparameter. Three values per knob times five knobs is 243 runs. Use grid only when the space is small and you want a tidy result.

Random search samples each knob independently. It almost always beats grid for the same budget because most hyperparameters do not matter much; random spreads your tries across the few that do. For most projects, random search with 30 to 100 trials is enough.

Bayesian optimization builds a surrogate model of the score surface and proposes the next configuration that balances exploration and exploitation. It is faster per trial in well-behaved spaces but adds overhead and can get stuck in plateaus.

Hyperband and ASHA train many configurations briefly, kill the worst ones, and train the survivors longer. This works very well for deep learning where partial-training curves are informative. The cost is implementation complexity.

Practical tips

Search over the log scale for learning rate and regularization. These parameters affect the model multiplicatively, and a uniform sample misses the interesting regions.

Tune one model at a time. Trying to compare random forests, XGBoost, and SVMs in one giant search wastes budget. Pick the family first, then tune.

Use cross-validation for small datasets, a fixed validation set for large ones. CV is more honest but expensive. A single split is fine when you have hundreds of thousands of rows.

Cap each trial’s wall-clock or step budget. Without a cap, a few pathological configurations will dominate your total time. Early termination is your friend.

Save every trial’s results, not just the best one. The full table tells you which knobs mattered, where the surface was flat, and which ranges to refine on the next pass.

Wrap-up

The right tuning strategy depends on how expensive a single training run is. Small models: random search, plenty of trials, done. Medium models: Bayesian optimization with a sensible budget. Large models: Hyperband or ASHA with aggressive early stopping. In all cases, the biggest lift comes from defining a sensible search space, not from picking the fanciest algorithm.