Gradient Boosting with XGBoost: A Practical Tutorial

What Is Gradient Boosting

Gradient boosting is an ensemble technique that builds models sequentially. Each new model corrects the errors of the combined ensemble so far. Unlike random forests, which build trees independently and average their predictions, gradient boosting adds trees one at a time, each focused on the residual mistakes.

The “gradient” part refers to how errors are corrected. At each step, the algorithm computes the gradient of the loss function with respect to the current predictions and fits a new tree to those gradients. This is essentially gradient descent in function space.

XGBoost (Extreme Gradient Boosting) is the most popular implementation. It adds regularization, parallel processing, and clever handling of missing values on top of the core algorithm.

Installing XGBoost

pip install xgboost

Your First XGBoost Model

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

# Load data
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

# Train model
model = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=4,
    learning_rate=0.1,
    random_state=42,
    eval_metric="logloss",
)
model.fit(X_train, y_train)

# Evaluate
y_pred = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(classification_report(y_test, y_pred, target_names=data.target_names))

This trains 100 trees, each with a maximum depth of 4. The learning_rate controls how much each tree contributes. Lower values need more trees but often produce better results.

How Gradient Boosting Builds Trees

The process works like this:

1. Start with a simple prediction, usually the mean of the target (regression) or the log-odds (classification).
2. Compute the residuals: the difference between actual values and current predictions.
3. Fit a shallow decision tree to predict those residuals.
4. Add the new tree’s predictions to the ensemble, scaled by the learning rate.
5. Repeat from step 2.

Each tree learns to fix what the previous ensemble got wrong. Early trees capture the broad patterns. Later trees handle edge cases and subtle interactions.

# Visualize the boosting process with a simple regression example
import numpy as np
import xgboost as xgb

np.random.seed(42)
X = np.sort(np.random.uniform(0, 10, 200)).reshape(-1, 1)
y = np.sin(X.ravel()) + np.random.normal(0, 0.2, 200)

# Train with different numbers of boosting rounds
for n_trees in [1, 5, 20, 100]:
    model = xgb.XGBRegressor(n_estimators=n_trees, max_depth=3, learning_rate=0.1)
    model.fit(X, y)
    pred = model.predict(X)
    mse = np.mean((y - pred) ** 2)
    print(f"Trees: {n_trees:3d}  MSE: {mse:.4f}")

You will see MSE drop sharply with the first few trees and then improve more slowly. This diminishing returns pattern is characteristic of boosting.

Key Hyperparameters

Learning Rate and Number of Trees

These two parameters are linked. A lower learning rate requires more trees to reach the same performance but generalizes better. A common strategy is to set the learning rate to 0.01-0.1 and use early stopping to find the right number of trees.

model = xgb.XGBClassifier(
    n_estimators=1000,       # Set high, early stopping will find the right number
    learning_rate=0.05,
    random_state=42,
    eval_metric="logloss",
    early_stopping_rounds=20,
)

model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    verbose=False,
)
print(f"Best iteration: {model.best_iteration}")
print(f"Best score: {model.best_score:.4f}")

Early stopping monitors validation performance and stops training when it has not improved for early_stopping_rounds consecutive rounds.

Tree Depth and Complexity

max_depth controls how deep each tree can grow. Shallow trees (depth 3-6) act as weak learners, which is what boosting wants. Deep trees can overfit to the residuals.

min_child_weight sets the minimum sum of instance weight in a child node. Higher values make the model more conservative.

model = xgb.XGBClassifier(
    max_depth=4,
    min_child_weight=3,
    gamma=0.1,              # Minimum loss reduction for a split
    random_state=42,
)

Regularization

XGBoost supports both L1 (reg_alpha) and L2 (reg_lambda) regularization on leaf weights. These penalize complex trees and reduce overfitting.

model = xgb.XGBClassifier(
    reg_alpha=0.1,   # L1 regularization
    reg_lambda=1.0,  # L2 regularization
    random_state=42,
)

Subsampling

Like random forests, XGBoost can subsample rows and columns at each boosting round. This adds randomness that reduces overfitting.

model = xgb.XGBClassifier(
    subsample=0.8,            # Use 80% of rows per tree
    colsample_bytree=0.8,     # Use 80% of features per tree
    random_state=42,
)

Feature Importance

XGBoost provides multiple ways to measure feature importance.

import pandas as pd

model.fit(X_train, y_train)

# Built-in feature importance (gain-based)
importance = model.feature_importances_
feature_names = data.feature_names

importance_df = pd.DataFrame({
    "feature": feature_names,
    "importance": importance,
}).sort_values("importance", ascending=False).head(10)

print(importance_df.to_string(index=False))

The default importance type is weight (number of times a feature appears in trees). You can switch to gain (average improvement in loss) or cover (average number of samples affected).

# Plot with XGBoost's built-in plotting
xgb.plot_importance(model, importance_type="gain", max_num_features=10)

Gain-based importance is generally more informative than weight-based importance because it measures how much each feature actually helps the model rather than just how often it is used.

Handling Missing Values

XGBoost handles missing values natively. During training, it learns the optimal direction (left or right) for missing values at each split. You do not need to impute missing values before training.

import numpy as np

# Introduce missing values
X_train_missing = X_train.copy()
mask = np.random.random(X_train_missing.shape) < 0.1
X_train_missing[mask] = np.nan

# XGBoost handles this automatically
model = xgb.XGBClassifier(n_estimators=100, random_state=42)
model.fit(X_train_missing, y_train)

Hyperparameter Tuning with Grid Search

from sklearn.model_selection import GridSearchCV

param_grid = {
    "max_depth": [3, 4, 5, 6],
    "learning_rate": [0.01, 0.05, 0.1],
    "subsample": [0.8, 1.0],
    "colsample_bytree": [0.8, 1.0],
    "reg_alpha": [0, 0.1],
    "reg_lambda": [1, 2],
}

grid_search = GridSearchCV(
    xgb.XGBClassifier(n_estimators=200, random_state=42, eval_metric="logloss"),
    param_grid,
    cv=5,
    scoring="f1",
    n_jobs=-1,
    verbose=1,
)

grid_search.fit(X_train, y_train)
print(f"Best params: {grid_search.best_params_}")
print(f"Best F1: {grid_search.best_score_:.3f}")

For larger search spaces, consider randomized search or Bayesian optimization with Optuna instead of exhaustive grid search.

XGBoost vs. LightGBM vs. CatBoost

XGBoost grows trees level-by-level (breadth-first). LightGBM grows trees leaf-by-level (depth-first), which is often faster on large datasets. CatBoost handles categorical features natively without manual encoding.

All three produce similar accuracy on most problems. Choose XGBoost when you want the most battle-tested option with extensive documentation. Choose LightGBM for speed on large datasets. Choose CatBoost when you have many categorical features.

Common Mistakes

Setting n_estimators too high without early stopping leads to overfitting. Always use early stopping or cross-validation to determine the right number of rounds.

Using deep trees (max_depth greater than 8) with boosting usually hurts. Boosting works best with weak learners. If each tree is too powerful, the ensemble overfits quickly.

Ignoring the learning rate and fixing it at 1.0 defeats the purpose of gradual learning. Start with 0.1 and decrease if validation performance plateaus.

Summary

Gradient boosting builds models sequentially, with each tree correcting the errors of the ensemble so far. XGBoost is the most popular implementation, offering regularization, early stopping, and native missing value handling. The most important hyperparameters are learning rate, number of trees, tree depth, and subsampling ratios. Use early stopping to avoid overfitting and feature importance to understand what your model learned.