Cross-Validation Strategies: K-Fold, Stratified, Time Series, and Nested CV

A single train-test split gives you one number. Change the random seed and that number shifts. Cross-validation gives you multiple estimates by rotating which portion of the data serves as the test set, producing a more stable and trustworthy measure of model performance. This guide covers every cross-validation strategy you will encounter in practice.

K-Fold Cross-Validation

K-fold splits the dataset into k equally sized folds. The model trains on k-1 folds and validates on the remaining fold. This repeats k times, with each fold serving as the validation set exactly once.

Fold 1: [VAL] [train] [train] [train] [train]
Fold 2: [train] [VAL] [train] [train] [train]
Fold 3: [train] [train] [VAL] [train] [train]
Fold 4: [train] [train] [train] [VAL] [train]
Fold 5: [train] [train] [train] [train] [VAL]

Final score = mean of 5 validation scores

from sklearn.model_selection import KFold, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
import numpy as np

X, y = make_classification(n_samples=1000, n_features=20,
                           n_informative=10, random_state=42)

model = RandomForestClassifier(n_estimators=100, random_state=42)

# Basic k-fold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=kf, scoring='accuracy')

print(f"Fold scores: {scores}")
print(f"Mean: {scores.mean():.4f} (+/- {scores.std():.4f})")

Choosing k

• k=5 is the most common default. Good balance between bias and variance.
• k=10 gives lower bias but higher variance and costs more compute.
• k=n (Leave-One-Out) is nearly unbiased but extremely expensive and has high variance.

A practical rule: use k=5 for quick experiments, k=10 for final evaluation.

Stratified K-Fold

Standard k-fold does not guarantee that each fold has the same class distribution as the original dataset. If your dataset has 5% positive examples, one fold might get 2% and another 8%. Stratified k-fold preserves the class ratio in every fold.

from sklearn.model_selection import StratifiedKFold

# Imbalanced dataset
X_imb, y_imb = make_classification(n_samples=1000, n_features=20,
                                    weights=[0.95, 0.05],
                                    random_state=42)

print(f"Overall positive rate: {y_imb.mean():.3f}")

# Standard k-fold -- class ratios vary across folds
kf = KFold(n_splits=5, shuffle=True, random_state=42)
for i, (train_idx, val_idx) in enumerate(kf.split(X_imb, y_imb)):
    print(f"  KFold {i+1}: val positive rate = {y_imb[val_idx].mean():.3f}")

print()

# Stratified k-fold -- class ratios preserved
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for i, (train_idx, val_idx) in enumerate(skf.split(X_imb, y_imb)):
    print(f"  StratifiedKFold {i+1}: val positive rate = {y_imb[val_idx].mean():.3f}")

# Use StratifiedKFold for scoring
scores = cross_val_score(model, X_imb, y_imb, cv=skf, scoring='f1')
print(f"\nStratified CV F1: {scores.mean():.4f} (+/- {scores.std():.4f})")

Always use stratified k-fold for classification tasks. Scikit-learn’s cross_val_score defaults to stratified k-fold for classifiers, but it is good practice to be explicit.

Repeated K-Fold

Running k-fold multiple times with different random splits reduces variance in the estimate.

from sklearn.model_selection import RepeatedStratifiedKFold

rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=3, random_state=42)
scores = cross_val_score(model, X, y, cv=rskf, scoring='accuracy')

print(f"15 fold scores (5-fold x 3 repeats)")
print(f"Mean: {scores.mean():.4f} (+/- {scores.std():.4f})")

The cost is 3x the compute of a single run, but you get a tighter confidence interval on the true performance.

Group K-Fold

When samples from the same group (e.g., same patient, same user) should not appear in both train and validation sets, use GroupKFold.

from sklearn.model_selection import GroupKFold

# Simulate data with groups (e.g., multiple samples per patient)
groups = np.array([i // 5 for i in range(1000)])  # 200 groups of 5

gkf = GroupKFold(n_splits=5)
scores = cross_val_score(model, X, y, cv=gkf, groups=groups, scoring='accuracy')
print(f"GroupKFold accuracy: {scores.mean():.4f} (+/- {scores.std():.4f})")

# Verify no group overlap
for fold, (train_idx, val_idx) in enumerate(gkf.split(X, y, groups)):
    train_groups = set(groups[train_idx])
    val_groups = set(groups[val_idx])
    overlap = train_groups & val_groups
    print(f"Fold {fold+1}: {len(val_groups)} val groups, overlap: {len(overlap)}")

Without group splitting, you get data leakage: the model sees examples from the same patient during training and validation, inflating the score.

Time Series Cross-Validation

Time series data has temporal ordering. Standard k-fold randomly mixes past and future data, which is leakage. Time series CV always trains on past data and validates on future data.

Fold 1: [TRAIN]  [VAL]   -----   -----   -----
Fold 2: [TRAIN] [TRAIN]  [VAL]   -----   -----
Fold 3: [TRAIN] [TRAIN] [TRAIN]  [VAL]   -----
Fold 4: [TRAIN] [TRAIN] [TRAIN] [TRAIN]  [VAL]

Training window expands, validation always follows.
No future data leaks into training.

from sklearn.model_selection import TimeSeriesSplit
from sklearn.linear_model import Ridge
import pandas as pd

# Simulate time series data
np.random.seed(42)
n = 1000
dates = pd.date_range('2020-01-01', periods=n, freq='D')
X_ts = np.column_stack([
    np.sin(np.arange(n) * 2 * np.pi / 365),
    np.random.randn(n),
    np.arange(n) / n
])
y_ts = 2 * X_ts[:, 0] + 0.5 * X_ts[:, 2] + np.random.randn(n) * 0.3

# Time series cross-validation
tscv = TimeSeriesSplit(n_splits=5)

model_ts = Ridge(alpha=1.0)
scores = cross_val_score(model_ts, X_ts, y_ts, cv=tscv, scoring='r2')

for i, score in enumerate(scores):
    train_idx, val_idx = list(tscv.split(X_ts))[i]
    print(f"Fold {i+1}: train={len(train_idx)}, val={len(val_idx)}, "
          f"R2={score:.4f}")

print(f"\nMean R2: {scores.mean():.4f}")

Custom Sliding Window

For a fixed-size training window instead of an expanding one:

class SlidingWindowCV:
    """Fixed-size sliding window for time series."""
    
    def __init__(self, train_size, val_size, step=None):
        self.train_size = train_size
        self.val_size = val_size
        self.step = step or val_size
    
    def split(self, X, y=None, groups=None):
        n = len(X)
        start = 0
        while start + self.train_size + self.val_size <= n:
            train_idx = np.arange(start, start + self.train_size)
            val_idx = np.arange(start + self.train_size,
                                start + self.train_size + self.val_size)
            yield train_idx, val_idx
            start += self.step
    
    def get_n_splits(self, X=None, y=None, groups=None):
        if X is None:
            return 0
        n = len(X)
        return max(0, (n - self.train_size - self.val_size) // self.step + 1)

sw_cv = SlidingWindowCV(train_size=500, val_size=100, step=100)
scores = cross_val_score(model_ts, X_ts, y_ts, cv=sw_cv, scoring='r2')
print(f"Sliding window R2: {scores.mean():.4f} (+/- {scores.std():.4f})")

Nested Cross-Validation

When you combine hyperparameter tuning with performance evaluation, a single loop of CV introduces bias: the best hyperparameters are chosen to maximize the validation score, which inflates the estimate. Nested CV uses two loops to give an unbiased estimate.

Outer loop (performance estimation):
For each outer fold:
  Inner loop (hyperparameter tuning):
    For each inner fold:
      Train model with candidate params
    Select best params based on inner CV score
  Train model with best params on outer training set
  Evaluate on outer test set

Final estimate = mean of outer test scores

from sklearn.model_selection import (
    cross_val_score, GridSearchCV, StratifiedKFold
)
from sklearn.svm import SVC

X, y = make_classification(n_samples=500, n_features=20, random_state=42)

# Inner CV: hyperparameter tuning
param_grid = {'C': [0.1, 1, 10], 'gamma': ['scale', 'auto']}
inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
grid_search = GridSearchCV(SVC(), param_grid, cv=inner_cv,
                           scoring='accuracy', n_jobs=-1)

# Outer CV: performance estimation
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
nested_scores = cross_val_score(grid_search, X, y, cv=outer_cv,
                                 scoring='accuracy')

print(f"Nested CV accuracy: {nested_scores.mean():.4f} "
      f"(+/- {nested_scores.std():.4f})")

# Compare with non-nested (biased) estimate
grid_search.fit(X, y)
print(f"Non-nested best CV: {grid_search.best_score_:.4f}  (optimistic)")

The non-nested score is typically 1-3% higher than the nested score. The nested score is the honest one.

Quick Reference

Key Takeaways

Use stratified k-fold as your default for classification. Use time series split whenever data has temporal ordering. Use group k-fold when samples are not independent. Use nested CV when you are tuning hyperparameters and need an honest performance estimate. The extra compute of proper cross-validation is always worth it compared to the cost of deploying a model whose true performance you do not know.