Hyperparameter Tuning Guide: Grid Search to Optuna

Every machine learning model has hyperparameters: settings you fix before training begins. Learning rate, tree depth, regularization strength, number of hidden units. Default values get you started, but tuning these values can lift performance from acceptable to production-grade. This guide walks through every major tuning strategy with working code.

Why Tuning Matters

Consider a random forest. With n_estimators=10 and max_depth=3, it underfits. With n_estimators=500 and max_depth=None, it might overfit. Somewhere in between lies the sweet spot. The difference in accuracy between default and tuned hyperparameters is often 2-5 percentage points, which can be the gap between a useful model and a useless one.

Grid Search

Grid search evaluates every combination of hyperparameter values you specify. It is exhaustive and guaranteed to find the best combination within the grid.

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=1000, n_features=20,
                           n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                     test_size=0.2,
                                                     random_state=42)

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, 20, None],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4]
}

grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring='f1',
    n_jobs=-1,
    verbose=1
)

grid_search.fit(X_train, y_train)

print(f"Best params: {grid_search.best_params_}")
print(f"Best CV F1:  {grid_search.best_score_:.4f}")
print(f"Test F1:     {grid_search.score(X_test, y_test):.4f}")

The problem with grid search is combinatorial explosion. The grid above has 3 x 4 x 3 x 3 = 108 combinations. With 5-fold CV, that is 540 model fits. Add one more hyperparameter with 4 values and you are at 2,160 fits. Grid search scales poorly.

Grid Search:              Random Search:
+---+---+---+             +---+---+---+
|   | x |   |             | x |   |   |
+---+---+---+             +---+---+---+
| x |   | x |             |   |   | x |
+---+---+---+             +---+---+---+
|   | x |   |             |   | x |   |
+---+---+---+             +---+---+---+

Grid: covers intersections   Random: covers more of each
only at predefined points    dimension independently

Random Search

Random search samples hyperparameter values from distributions you define. It does not try every combination. Instead, it picks random points in the search space.

from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint, uniform

param_distributions = {
    'n_estimators': randint(50, 500),
    'max_depth': randint(3, 30),
    'min_samples_split': randint(2, 20),
    'min_samples_leaf': randint(1, 10),
    'max_features': uniform(0.1, 0.9)
}

random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_distributions,
    n_iter=50,           # try 50 random combinations
    cv=5,
    scoring='f1',
    n_jobs=-1,
    random_state=42,
    verbose=1
)

random_search.fit(X_train, y_train)

print(f"Best params: {random_search.best_params_}")
print(f"Best CV F1:  {random_search.best_score_:.4f}")

Bergstra and Bengio (2012) showed that random search finds good hyperparameters faster than grid search in most cases. The reason is that different hyperparameters have different importance. If learning rate matters far more than batch size, grid search wastes most of its budget varying batch size at each learning rate. Random search explores more unique values of the important parameter.

Bayesian Optimization

Bayesian optimization builds a probabilistic model of the objective function and uses it to choose the next point to evaluate. It balances exploration (trying new regions) with exploitation (focusing on promising regions).

# Using scikit-optimize
from skopt import BayesSearchCV
from skopt.space import Integer, Real

bayes_search = BayesSearchCV(
    RandomForestClassifier(random_state=42),
    {
        'n_estimators': Integer(50, 500),
        'max_depth': Integer(3, 30),
        'min_samples_split': Integer(2, 20),
        'min_samples_leaf': Integer(1, 10),
        'max_features': Real(0.1, 1.0)
    },
    n_iter=50,
    cv=5,
    scoring='f1',
    n_jobs=-1,
    random_state=42
)

bayes_search.fit(X_train, y_train)

print(f"Best params: {bayes_search.best_params_}")
print(f"Best CV F1:  {bayes_search.best_score_:.4f}")

The surrogate model (typically a Gaussian Process or Tree-structured Parzen Estimator) learns which regions of the search space yield good results. Each new trial is chosen to maximize expected improvement. This makes Bayesian optimization converge faster than random search, especially when evaluations are expensive.

Optuna

Optuna is the modern choice for hyperparameter tuning. It uses a Tree-structured Parzen Estimator (TPE) by default, supports pruning of unpromising trials, handles conditional parameters, and provides built-in visualization.

import optuna
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 50, 500),
        'max_depth': trial.suggest_int('max_depth', 3, 30),
        'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),
        'max_features': trial.suggest_float('max_features', 0.1, 1.0),
    }
    
    model = RandomForestClassifier(**params, random_state=42)
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring='f1')
    return scores.mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100, show_progress_bar=True)

print(f"Best trial value: {study.best_trial.value:.4f}")
print(f"Best params: {study.best_trial.params}")

Pruning Unpromising Trials

Optuna can stop bad trials early using pruning. This saves compute by killing runs that are clearly underperforming.

from sklearn.model_selection import StratifiedKFold
import numpy as np

def objective_with_pruning(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 50, 500),
        'max_depth': trial.suggest_int('max_depth', 3, 30),
        'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
        'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),
    }
    
    model = RandomForestClassifier(**params, random_state=42)
    skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
    
    scores = []
    for step, (train_idx, val_idx) in enumerate(skf.split(X_train, y_train)):
        model.fit(X_train[train_idx], y_train[train_idx])
        score = model.score(X_train[val_idx], y_train[val_idx])
        scores.append(score)
        
        # Report intermediate value for pruning
        trial.report(np.mean(scores), step)
        
        # Prune if this trial is not promising
        if trial.should_prune():
            raise optuna.exceptions.TrialPruned()
    
    return np.mean(scores)

study = optuna.create_study(
    direction='maximize',
    pruner=optuna.pruners.MedianPruner(n_warmup_steps=2)
)
study.optimize(objective_with_pruning, n_trials=100)

Conditional Hyperparameters

Optuna handles parameters that only apply in certain configurations.

def objective_conditional(trial):
    classifier_name = trial.suggest_categorical('classifier', ['rf', 'svm', 'xgb'])
    
    if classifier_name == 'rf':
        from sklearn.ensemble import RandomForestClassifier
        params = {
            'n_estimators': trial.suggest_int('rf_n_estimators', 50, 300),
            'max_depth': trial.suggest_int('rf_max_depth', 3, 20),
        }
        model = RandomForestClassifier(**params, random_state=42)
        
    elif classifier_name == 'svm':
        from sklearn.svm import SVC
        params = {
            'C': trial.suggest_float('svm_C', 1e-3, 100, log=True),
            'kernel': trial.suggest_categorical('svm_kernel', ['rbf', 'linear']),
        }
        model = SVC(**params, random_state=42)
        
    else:
        from xgboost import XGBClassifier
        params = {
            'n_estimators': trial.suggest_int('xgb_n_estimators', 50, 300),
            'learning_rate': trial.suggest_float('xgb_lr', 0.01, 0.3, log=True),
            'max_depth': trial.suggest_int('xgb_max_depth', 3, 10),
        }
        model = XGBClassifier(**params, random_state=42,
                               use_label_encoder=False, eval_metric='logloss')
    
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring='f1')
    return scores.mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective_conditional, n_trials=100)

Visualization

# Built-in plots
optuna.visualization.plot_optimization_history(study).show()
optuna.visualization.plot_param_importances(study).show()
optuna.visualization.plot_contour(study, params=['n_estimators', 'max_depth']).show()

Comparison Table

Practical Tips

1. Start with random search. It gives you a baseline in minutes. Switch to Optuna only if you need to squeeze out more performance.

2. Use log scales for learning rates and regularization. These parameters span orders of magnitude. suggest_float('lr', 1e-5, 1e-1, log=True) explores the space much more effectively than a linear range.

3. Fix a random seed for reproducibility. Tuning results should be reproducible. Set seeds in both the search and the model.

4. Budget your compute. Decide upfront how many trials you can afford. If training takes 10 minutes, 100 trials is 16 hours. Plan accordingly.

5. Watch for overfitting the validation set. Running hundreds of trials on the same CV folds is a form of information leakage. Use a held-out test set for final evaluation.

Key Takeaways

Grid search is simple but scales poorly. Random search gives better results per trial in most cases. Bayesian optimization and Optuna learn from previous trials, converging faster on expensive problems. Optuna’s pruning and conditional parameter support make it the best choice for serious tuning work. Whatever method you choose, always evaluate your final model on a held-out test set that was never used during tuning.