Cross-Validation Strategies: K-Fold, Stratified, and Time Series

Why Cross-Validation

A single train-test split gives you one number. That number depends heavily on which rows ended up in each set. Change the random seed, and accuracy might swing by several percentage points. Cross-validation gives you multiple estimates by rotating which portion of the data serves as the test set. The result is a more stable and trustworthy measure of model performance.

Cross-validation also helps you detect overfitting early. If your model scores 98% on training folds but 72% on validation folds, you know it is memorizing rather than learning.

K-Fold Cross-Validation

K-fold cross-validation splits the dataset into k equally sized folds. The model trains on k-1 folds and validates on the remaining fold. This process repeats k times, with each fold serving as the validation set exactly once. The final metric is the average across all k rounds.

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, random_state=42)
scores = cross_val_score(model, X, y, cv=5, scoring="accuracy")

print(f"Fold scores: {scores}")
print(f"Mean accuracy: {scores.mean():.3f} (+/- {scores.std():.3f})")

Choosing K

The most common choices are 5 and 10. With k=5, each fold contains 20% of the data. Training happens on 80%, which is close to a typical train-test split. With k=10, you get more precise estimates but training takes twice as long.

Small datasets benefit from higher k values because each fold retains more training data. With only 200 rows and k=5, each training set has 160 rows, which might be too few. Using k=10 gives you 180 training rows per fold.

Very large datasets can afford k=3 or even k=2 because there is enough data in each fold to get stable estimates without spending compute on many rounds.

How It Works Under the Hood

from sklearn.model_selection import KFold

kf = KFold(n_splits=5, shuffle=True, random_state=42)

for fold, (train_idx, val_idx) in enumerate(kf.split(X)):
    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

    model.fit(X_train, y_train)
    score = model.score(X_val, y_val)
    print(f"Fold {fold}: {score:.3f}")

Setting shuffle=True randomizes the order before splitting. This is important when the data has any inherent ordering, like rows sorted by date or by class label.

Stratified K-Fold

Standard k-fold can produce folds where the class distribution is uneven. If your dataset has 90% negative and 10% positive examples, a random split might give one fold 15% positives and another 5%. This makes fold-level scores unreliable.

Stratified k-fold ensures that each fold has approximately the same class distribution as the full dataset. Every fold gets close to 10% positives.

from sklearn.model_selection import StratifiedKFold, cross_val_score

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf, scoring="f1")

print(f"Stratified F1 scores: {scores}")
print(f"Mean F1: {scores.mean():.3f} (+/- {scores.std():.3f})")

When to Use Stratified K-Fold

Use stratified k-fold whenever you have a classification problem, especially with imbalanced classes. It is such a good default that scikit-learn uses it automatically when you pass an integer to cv in cross_val_score for classifiers.

For regression problems, stratified splitting is less common, but you can stratify on binned target values if the target distribution is highly skewed.

from sklearn.model_selection import StratifiedKFold
import numpy as np

# Stratify regression targets by binning
y_binned = pd.qcut(y, q=5, labels=False)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

for train_idx, val_idx in skf.split(X, y_binned):
    # Use original y for training, not the binned version
    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

Time Series Cross-Validation

Standard k-fold is wrong for time series data. It lets the model train on future data and predict the past, which is data leakage. In production, your model never has access to future data, so your validation scheme should not either.

Time series cross-validation uses an expanding window. The training set grows over time, and the validation set is always the next time period after training.

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5)

for fold, (train_idx, val_idx) in enumerate(tscv.split(X)):
    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

    model.fit(X_train, y_train)
    score = model.score(X_val, y_val)
    print(f"Fold {fold}: train size={len(train_idx)}, val size={len(val_idx)}, score={score:.3f}")

The Expanding Window Pattern

With 5 splits on 600 rows, the folds look like this:

• Fold 0: train on rows 0-99, validate on rows 100-199
• Fold 1: train on rows 0-199, validate on rows 200-299
• Fold 2: train on rows 0-299, validate on rows 300-399
• Fold 3: train on rows 0-399, validate on rows 400-499
• Fold 4: train on rows 0-499, validate on rows 500-599

Notice that training sets get larger with each fold. Early folds have less training data and may show worse performance. This is expected and reflects reality: a new model has less history to learn from.

Adding a Gap

Sometimes there is a delay between when data is collected and when predictions are needed. A sales forecast model might need to predict 7 days ahead. You can add a gap between training and validation.

from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits=5, gap=7)

for train_idx, val_idx in tscv.split(X):
    print(f"Train: {train_idx[0]}-{train_idx[-1]}, Val: {val_idx[0]}-{val_idx[-1]}")

The gap parameter skips rows between the end of training and the start of validation, preventing leakage from autocorrelated features.

Comparing Strategies Side by Side

Here is a function that runs all three strategies and prints results.

from sklearn.model_selection import (
    KFold, StratifiedKFold, TimeSeriesSplit, cross_val_score
)
from sklearn.ensemble import GradientBoostingClassifier

def compare_cv_strategies(X, y, model=None, scoring="accuracy"):
    if model is None:
        model = GradientBoostingClassifier(n_estimators=100, random_state=42)

    strategies = {
        "KFold": KFold(n_splits=5, shuffle=True, random_state=42),
        "StratifiedKFold": StratifiedKFold(n_splits=5, shuffle=True, random_state=42),
        "TimeSeriesSplit": TimeSeriesSplit(n_splits=5),
    }

    for name, cv in strategies.items():
        scores = cross_val_score(model, X, y, cv=cv, scoring=scoring)
        print(f"{name:20s} -> {scores.mean():.3f} (+/- {scores.std():.3f})")

compare_cv_strategies(X, y)

Nested Cross-Validation

When you use cross-validation to both tune hyperparameters and estimate performance, you get optimistically biased results. Nested cross-validation solves this with two loops: an outer loop estimates performance and an inner loop tunes hyperparameters.

from sklearn.model_selection import GridSearchCV, cross_val_score

param_grid = {"n_estimators": [50, 100, 200], "max_depth": [3, 5, 7]}

inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

grid_search = GridSearchCV(model, param_grid, cv=inner_cv, scoring="f1")
nested_scores = cross_val_score(grid_search, X, y, cv=outer_cv, scoring="f1")

print(f"Nested CV F1: {nested_scores.mean():.3f} (+/- {nested_scores.std():.3f})")

This is computationally expensive but gives you an honest estimate of how well your tuned model will perform on truly unseen data.

Quick Reference

Use k-fold as your default for tabular data with no temporal component. Use stratified k-fold for classification tasks, especially with class imbalance. Use time series split whenever rows have a meaningful time ordering. Use nested CV when you need to tune hyperparameters and report unbiased performance in the same experiment.

Always report the mean and standard deviation of scores across folds. A model with 85% mean accuracy and 1% standard deviation is far more trustworthy than one with 87% mean accuracy and 8% standard deviation.