ML Bias Variance Tradeoff

The bias variance tradeoff is one of those topics that comes up in every ML course and somehow remains fuzzy until you have shipped a few models. This post explains it in plain language, shows how to spot each problem in your training curves, and lists the actual moves you can make in response.

What bias and variance really are

Imagine training the same model on many different random samples of training data. Each run gives slightly different predictions. Bias is how far the average prediction is from the true value. Variance is how much the predictions jiggle around that average from run to run.

High bias means the model is too simple to capture the underlying pattern. It is wrong in a consistent way. High variance means the model is sensitive to the particular training set. It is right on average but unstable.

Total expected error breaks roughly into bias squared plus variance plus irreducible noise. You can trade bias for variance and back, but the sum has a floor set by the noise.

Mental model

Picture a dartboard. Low bias and low variance means all darts cluster around the bullseye. Low bias and high variance means darts scatter widely around the bullseye on average. High bias and low variance means darts cluster tightly but in the wrong spot. High bias and high variance means darts are scattered and off-center.

            low variance      high variance
low bias    [tight bullseye]   [scattered, centered]
high bias   [tight off-center] [scattered, off-center]

Hands-on example

A clear way to see both is fitting polynomials of increasing degree to a small dataset.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

rng = np.random.default_rng(0)
X = np.linspace(0, 1, 30).reshape(-1, 1)
y = np.sin(2 * np.pi * X.ravel()) + rng.normal(0, 0.2, 30)

for d in [1, 3, 9, 20]:
    m = make_pipeline(PolynomialFeatures(d), LinearRegression())
    cv = cross_val_score(m, X, y, scoring="neg_mean_squared_error", cv=5).mean()
    print(f"degree {d}: cv mse = {-cv:.3f}")

Degree 1 is too rigid: high bias, low variance. Degree 20 chases noise: low bias, high variance. Degree 3 hits the sweet spot. You can read the same story off learning curves: training and validation error both high means high bias; large gap between them means high variance.

Trade-offs

Reducing bias usually requires a more flexible model: more features, more parameters, deeper trees, a richer kernel. The cost is higher variance and more data needed to fit reliably.

Reducing variance usually requires regularization, more data, or ensembling. Lasso and ridge shrink coefficients. Dropout in neural networks averages over thinned networks. Bagging averages over models trained on resampled data.

In the deep learning era, large overparameterized models break the classic picture. They have enough capacity to memorize training data yet generalize well, especially with the right optimizer and data augmentation. The tradeoff still exists, but the location of the sweet spot has moved.

Practical tips

Plot learning curves. Train your model on increasing fractions of the data and plot training and validation error. The shape tells you whether you are bias-limited (both high), variance-limited (large gap), or near-optimal (both low and close).

If you are bias-limited, add features or capacity. Engineer new inputs, switch to a more flexible model, increase tree depth, or add hidden layers.

If you are variance-limited, get more data first. Then add regularization. For trees, prune or use bagging. For neural networks, add weight decay or dropout. For linear models, increase alpha in ridge.

Hold out a clean test set you never tune on. Cross-validation guides selection, the held-out set tells you what production will see.

Beware the temptation to keep tweaking. Once you are near the noise floor, additional engineering work returns less. Spend that time on data quality or downstream metrics instead.

Wrap-up

Bias and variance are two sides of the error budget. You can spend on one or the other, but the irreducible noise sets a hard floor. The skill is in diagnosing which side dominates and matching your next move to the diagnosis. Learning curves are the cheapest tool to do that, and they should be the first plot you draw on any new project.