ML Ensemble Methods Beyond Random Forests

A single model has a ceiling. It learns one set of patterns, makes one type of error, and has one set of blind spots. Ensemble methods combine multiple models to cancel out individual weaknesses, and they consistently win machine learning competitions for a reason: they work. While random forests and gradient boosting are the most well-known ensemble techniques, stacking, voting, and blending offer additional ways to squeeze performance out of your models.

Why Ensembles Work

The mathematical basis is straightforward. If you have multiple models that make independent errors, averaging their predictions reduces the overall error. Consider three classifiers, each with 70% accuracy on a binary task. If they make errors independently, a majority vote achieves:

P(majority correct) = P(all 3 correct) + P(exactly 2 correct)
                    = 0.7^3 + 3 * 0.7^2 * 0.3
                    = 0.343 + 0.441
                    = 0.784

The ensemble is more accurate than any individual model. The key requirement is that the models make different errors — correlated errors do not cancel out.

Bagging vs. Boosting vs. Stacking

These three strategies represent fundamentally different approaches to combining models.

Bagging (Bootstrap Aggregating) trains identical model architectures on random subsets of data. Random Forests are the canonical example. Each tree sees a different data sample and feature subset, creating diversity through data variation.

Boosting trains models sequentially, where each new model focuses on the examples the previous models got wrong. Gradient Boosting (XGBoost, LightGBM, CatBoost) is the most popular variant. Diversity comes from error correction.

Stacking trains different model types on the same data, then trains a meta-model to combine their predictions. Diversity comes from architectural differences between base models.

Voting Ensembles

The simplest ensemble: multiple models vote on the prediction. For classification, use majority vote. For regression, use the average.

from sklearn.ensemble import (
    VotingClassifier, RandomForestClassifier, GradientBoostingClassifier,
)
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Hard voting: majority wins
hard_voting = VotingClassifier(
    estimators=[
        ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
        ("gb", GradientBoostingClassifier(n_estimators=100, random_state=42)),
        ("lr", LogisticRegression(max_iter=1000)),
        ("svm", SVC(kernel="rbf", probability=True)),
    ],
    voting="hard",
)

# Soft voting: average predicted probabilities
soft_voting = VotingClassifier(
    estimators=[
        ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
        ("gb", GradientBoostingClassifier(n_estimators=100, random_state=42)),
        ("lr", LogisticRegression(max_iter=1000)),
        ("svm", SVC(kernel="rbf", probability=True)),
    ],
    voting="soft",
)

# Compare individual models vs. ensemble
models = {
    "Random Forest": RandomForestClassifier(n_estimators=100, random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(n_estimators=100, random_state=42),
    "Logistic Regression": LogisticRegression(max_iter=1000),
    "Hard Voting": hard_voting,
    "Soft Voting": soft_voting,
}

for name, model in models.items():
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring="accuracy")
    print(f"{name}: {scores.mean():.4f} (+/- {scores.std():.4f})")

Soft voting typically outperforms hard voting because it uses probability estimates rather than discrete predictions. A model that is 90% confident contributes more than one that is 51% confident.

Weighted Voting

Not all models deserve equal say. Weight better models higher.

from sklearn.metrics import accuracy_score

def find_optimal_weights(models, X_val, y_val):
    """Find weights proportional to validation accuracy."""
    weights = []
    for name, model in models:
        y_pred = model.predict(X_val)
        acc = accuracy_score(y_val, y_pred)
        weights.append(acc)
        print(f"{name}: accuracy = {acc:.4f}")

    # Normalize weights
    total = sum(weights)
    weights = [w / total for w in weights]
    return weights

# Use with VotingClassifier
weighted_voting = VotingClassifier(
    estimators=[
        ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
        ("gb", GradientBoostingClassifier(n_estimators=100, random_state=42)),
        ("lr", LogisticRegression(max_iter=1000)),
    ],
    voting="soft",
    weights=[0.4, 0.4, 0.2],  # Weights from validation performance
)

Stacking

Stacking goes beyond voting by training a meta-model (also called a blender) that learns how to optimally combine base model predictions.

from sklearn.ensemble import StackingClassifier

stacking = StackingClassifier(
    estimators=[
        ("rf", RandomForestClassifier(n_estimators=100, random_state=42)),
        ("gb", GradientBoostingClassifier(n_estimators=100, random_state=42)),
        ("svm", SVC(kernel="rbf", probability=True)),
    ],
    final_estimator=LogisticRegression(max_iter=1000),
    cv=5,  # Use cross-validation to generate meta-features
    stack_method="predict_proba",  # Use probabilities, not hard predictions
    passthrough=False,  # Set True to also pass original features to meta-model
)

scores = cross_val_score(stacking, X_train, y_train, cv=5, scoring="accuracy")
print(f"Stacking: {scores.mean():.4f} (+/- {scores.std():.4f})")

The cv=5 parameter is critical. It uses cross-validation to generate out-of-fold predictions for the meta-model’s training data, preventing the meta-model from simply memorizing which base model is always right on the training set.

Manual Stacking for More Control

When you need custom logic or want to understand what is happening inside the stack:

import numpy as np
from sklearn.model_selection import KFold

def manual_stacking(
    base_models: list,
    meta_model,
    X_train, y_train,
    X_test,
    n_folds: int = 5,
):
    """Build stacking ensemble with explicit cross-validation."""
    kf = KFold(n_splits=n_folds, shuffle=True, random_state=42)
    n_train = X_train.shape[0]
    n_test = X_test.shape[0]
    n_models = len(base_models)

    # Out-of-fold predictions for training the meta-model
    meta_train = np.zeros((n_train, n_models))
    # Average test predictions across folds
    meta_test = np.zeros((n_test, n_models))

    for i, model in enumerate(base_models):
        test_preds_folds = np.zeros((n_test, n_folds))

        for fold, (train_idx, val_idx) in enumerate(kf.split(X_train)):
            X_fold_train = X_train[train_idx]
            y_fold_train = y_train[train_idx]
            X_fold_val = X_train[val_idx]

            model.fit(X_fold_train, y_fold_train)

            # Out-of-fold predictions
            meta_train[val_idx, i] = model.predict(X_fold_val)

            # Test predictions for this fold
            test_preds_folds[:, fold] = model.predict(X_test)

        # Average test predictions across folds
        meta_test[:, i] = test_preds_folds.mean(axis=1)

    # Train meta-model on out-of-fold predictions
    meta_model.fit(meta_train, y_train)
    final_predictions = meta_model.predict(meta_test)

    return final_predictions, meta_model

Choosing Base Models for Diversity

The ensemble’s strength depends on model diversity. Combining five random forests is nearly useless — they make the same errors. Combine models with fundamentally different learning strategies:

Model Type	Strengths	Weaknesses
Linear models	Simple boundaries, fast	Cannot capture nonlinear patterns
Decision trees	Nonlinear, interpretable	Overfitting, unstable
SVMs	Strong with margins	Slow on large datasets
KNN	No assumptions about data	Slow at prediction time
Neural networks	Complex patterns	Needs large datasets

A good ensemble picks one model from each “family” of learning algorithms. Three models from different families typically outperform ten models from the same family.

When Ensembles Are Not Worth It

Ensembles add complexity, training time, and inference latency. They are not always the right choice.

Skip ensembles when: a single model already meets your accuracy requirements, inference latency is critical (each additional model adds prediction time), model interpretability is a requirement (ensembles are harder to explain), or you are still in the feature engineering phase (better features beat better ensembles).

Use ensembles when: you have hit a performance plateau with single models, the cost of prediction errors is high, you have diverse model types that make different errors, and you are in a competition or production scenario where every fraction of a percent matters.

Summary

Ensemble methods combine the strengths of multiple models to achieve better predictions than any single model. Voting ensembles are the simplest to implement — start there. Move to stacking when you need the meta-model to learn optimal combination weights. Always ensure diversity among base models by choosing different algorithm families. Measure the improvement an ensemble provides against the added complexity, and use ensembles only when the accuracy gain justifies the cost.