Parameters are learned from data (weights, split thresholds). Hyperparameters are chosen before training and control the learning process: learning rate, tree depth, regularisation strength, number of neighbours. Good hyperparameters can be the difference between a mediocre and an excellent model. Tuning them efficiently — without overfitting the validation set — is a core skill.
The tuning problem#
We want
This objective is expensive (each evaluation trains a model), noisy, non-differentiable, and often mixes continuous, integer and categorical dimensions. It is a black-box optimisation problem.
Grid search#
Evaluate every combination on a predefined grid. Exhaustive and easy to parallelise, but the cost grows exponentially with the number of hyperparameters: 5 values for each of 6 hyperparameters = 15,625 runs.
Random search#
Sample configurations at random from specified distributions. Bergstra and Bengio (2012) showed that random search is usually far more efficient than grid search. The reason: typically only a few hyperparameters really matter. With a 3×3 grid over two hyperparameters, you test only 3 distinct values of the important one. With 9 random samples, you test 9 distinct values of each.
Bayesian optimisation#
Random search ignores what it has learned. Bayesian optimisation builds a surrogate model of the objective from past trials and chooses the next configuration by maximising an acquisition function that balances exploring uncertain regions and exploiting promising ones.
- Gaussian-process surrogates with Expected Improvement:
where $f^*$ is the best value so far.
- Tree-structured Parzen Estimators (TPE), used by Optuna and Hyperopt, model the densities of good and bad configurations and handle conditional, mixed-type spaces well.
Bayesian methods typically find good configurations in fewer trials — valuable when each trial takes hours.
Multi-fidelity methods: stop bad trials early#
Most configurations are clearly bad after a small fraction of the budget. Successive halving trains many configurations with a small budget (few epochs, a data subset), keeps the best fraction (e.g. top third), gives them more budget, and repeats. Hyperband runs successive halving with several trade-offs between breadth and depth. ASHA is an asynchronous version for parallel clusters. Combining Bayesian sampling with early stopping (e.g. BOHB, or Optuna with pruners) is the modern standard.
Practical example with Optuna#
import optuna
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, StratifiedKFold, train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
X, y = load_breast_cancer(return_X_y=True)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
cv = StratifiedKFold(5, shuffle=True, random_state=0)
def objective(trial):
params = {
"learning_rate": trial.suggest_float("learning_rate", 1e-3, 0.3, log=True),
"max_leaf_nodes": trial.suggest_int("max_leaf_nodes", 4, 64, log=True),
"min_samples_leaf": trial.suggest_int("min_samples_leaf", 5, 100, log=True),
"l2_regularization": trial.suggest_float("l2_regularization", 1e-6, 10, log=True),
"max_iter": 400,
}
model = HistGradientBoostingClassifier(**params, random_state=0)
return cross_val_score(model, X_dev, y_dev, cv=cv, scoring="roc_auc").mean()
study = optuna.create_study(direction="maximize", sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=60, show_progress_bar=False)
print("best CV AUC:", round(study.best_value, 4), "\nbest params:", study.best_params)
final = HistGradientBoostingClassifier(**study.best_params, max_iter=400, random_state=0).fit(X_dev, y_dev)
print("held-out test accuracy:", round(final.score(X_test, y_test), 4))Avoiding validation overfitting#
What to tune (and what not to)#
| Model | High-impact hyperparameters |
|---|---|
| Logistic / linear | Regularisation strength ($C$ or $\alpha$), penalty type |
| SVM (RBF) | $C$, $\gamma$ |
| Random forest | max_features, min_samples_leaf (trees: just "enough") |
| Gradient boosting | learning rate + number of trees (early stopping), depth/leaves, min child samples, subsampling |
| Neural networks | learning rate (by far), batch size, weight decay, schedule, architecture width/depth |
Tune the few that matter; fix the rest at sensible defaults. And remember: hyperparameter tuning usually gives smaller gains than better data and features.
Reproducibility#
Log every trial's configuration, seed, code version and score (Optuna, MLflow or Weights & Biases). Report the search space and budget in papers — a method tuned with 500 trials compared against a baseline tuned with 10 is not a fair comparison.