Gradient Boosting Explained: XGBoost vs LightGBM vs CatBoost

Gradient boosting is the most dominant algorithm for structured/tabular data. It wins Kaggle competitions, powers production recommendation systems, and consistently outperforms other approaches on medium-sized datasets. Three libraries dominate the space: XGBoost, LightGBM, and CatBoost. This article explains how gradient boosting works and provides a hands-on comparison of all three.

How Gradient Boosting Works

Gradient boosting builds an ensemble of weak learners (usually decision trees) sequentially. Each tree is trained to predict the residual errors of the ensemble so far.

Step 1: Start with initial prediction (e.g., mean of target)
Step 2: Compute residuals = actual - predicted
Step 3: Fit a small tree to the residuals
Step 4: Update predictions: new_pred = old_pred + learning_rate * tree_pred
Step 5: Repeat steps 2-4 for N rounds

Final prediction = initial + lr*tree1 + lr*tree2 + ... + lr*treeN

The “gradient” refers to the fact that residuals are the negative gradient of the loss function. This makes the algorithm equivalent to gradient descent in function space, allowing it to optimize any differentiable loss.

Setup

# Install all three
# pip install xgboost lightgbm catboost

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import mean_squared_error, accuracy_score
import time

# Regression dataset
housing = fetch_california_housing()
X, y = housing.data, housing.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

XGBoost

XGBoost (Extreme Gradient Boosting) was released in 2016 and quickly became the default. It introduced regularized boosting, column sampling, and efficient handling of sparse data.

import xgboost as xgb

# Train XGBoost
xgb_model = xgb.XGBRegressor(
    n_estimators=500,
    max_depth=6,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.1,         # L1 regularization
    reg_lambda=1.0,        # L2 regularization
    random_state=42,
    early_stopping_rounds=20,
    eval_metric='rmse'
)

start = time.time()
xgb_model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    verbose=False
)
xgb_time = time.time() - start

y_pred_xgb = xgb_model.predict(X_test)
xgb_rmse = np.sqrt(mean_squared_error(y_test, y_pred_xgb))
print(f"XGBoost RMSE: {xgb_rmse:.4f} | Time: {xgb_time:.2f}s")

XGBoost splits nodes using a pre-sorted or histogram-based algorithm. It builds trees level by level (breadth-first). Key parameters:

• max_depth: controls tree complexity
• learning_rate: shrinkage factor (lower = more trees needed)
• subsample: fraction of rows used per tree
• colsample_bytree: fraction of columns used per tree
• reg_alpha, reg_lambda: L1 and L2 regularization on leaf weights

LightGBM

LightGBM, released by Microsoft in 2017, introduced two innovations: Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB). It builds trees leaf-wise instead of level-wise, which often produces better accuracy with fewer splits.

import lightgbm as lgb

lgb_model = lgb.LGBMRegressor(
    n_estimators=500,
    max_depth=-1,          # no limit, controlled by num_leaves
    num_leaves=31,         # key parameter for LightGBM
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.1,
    reg_lambda=1.0,
    random_state=42,
    n_jobs=-1
)

start = time.time()
lgb_model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    callbacks=[lgb.early_stopping(20), lgb.log_evaluation(0)]
)
lgb_time = time.time() - start

y_pred_lgb = lgb_model.predict(X_test)
lgb_rmse = np.sqrt(mean_squared_error(y_test, y_pred_lgb))
print(f"LightGBM RMSE: {lgb_rmse:.4f} | Time: {lgb_time:.2f}s")

LightGBM’s leaf-wise growth finds the leaf with the highest gain and splits it, regardless of depth. This is more efficient but can overfit on small datasets. Control overfitting with num_leaves (not max_depth).

Level-wise (XGBoost):          Leaf-wise (LightGBM):
     [root]                        [root]
    /      \                      /      \
  [A]      [B]                  [A]      [B]
 /  \    /  \                /  \
[C] [D] [E] [F]             [C] [D]
                                  /  \
All nodes at each depth           [E]  [F]
split before going deeper.
                            Splits the highest-gain
                            leaf first, goes deeper
                            on one side.

CatBoost

CatBoost, by Yandex (2017), was designed for datasets with many categorical features. It handles them natively without manual encoding, using a technique called ordered target statistics.

from catboost import CatBoostRegressor

cat_model = CatBoostRegressor(
    iterations=500,
    depth=6,
    learning_rate=0.1,
    l2_leaf_reg=3.0,
    random_seed=42,
    verbose=0,
    early_stopping_rounds=20
)

start = time.time()
cat_model.fit(
    X_train, y_train,
    eval_set=(X_test, y_test),
    verbose=0
)
cat_time = time.time() - start

y_pred_cat = cat_model.predict(X_test)
cat_rmse = np.sqrt(mean_squared_error(y_test, y_pred_cat))
print(f"CatBoost RMSE: {cat_rmse:.4f} | Time: {cat_time:.2f}s")

Native Categorical Feature Handling

CatBoost’s biggest advantage is handling categoricals without preprocessing.

import pandas as pd
from catboost import CatBoostClassifier, Pool

# Example with categorical features
data = pd.DataFrame({
    'city': ['NYC', 'LA', 'NYC', 'Chicago', 'LA', 'Chicago'] * 100,
    'category': ['A', 'B', 'C', 'A', 'B', 'C'] * 100,
    'price': np.random.randn(600),
    'target': np.random.randint(0, 2, 600)
})

cat_features = ['city', 'category']
X = data.drop('target', axis=1)
y = data['target']

train_pool = Pool(X, y, cat_features=cat_features)

model = CatBoostClassifier(iterations=200, verbose=0, random_seed=42)
model.fit(train_pool)

# No encoding needed -- CatBoost handles it internally
# using ordered target statistics to prevent target leakage

Side-by-Side Comparison

print(f"\n{'Library':<12} {'RMSE':<10} {'Time (s)':<10}")
print("-" * 32)
print(f"{'XGBoost':<12} {xgb_rmse:<10.4f} {xgb_time:<10.2f}")
print(f"{'LightGBM':<12} {lgb_rmse:<10.4f} {lgb_time:<10.2f}")
print(f"{'CatBoost':<12} {cat_rmse:<10.4f} {cat_time:<10.2f}")

Classification Example

All three work for classification with minimal changes.

from sklearn.datasets import make_classification

X_cls, y_cls = make_classification(n_samples=5000, n_features=20,
                                    n_informative=15, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X_cls, y_cls,
                                            test_size=0.2, random_state=42)

# XGBoost classification
xgb_cls = xgb.XGBClassifier(n_estimators=200, max_depth=6,
                              learning_rate=0.1, random_state=42,
                              eval_metric='logloss')
xgb_cls.fit(X_tr, y_tr, eval_set=[(X_te, y_te)], verbose=False)

# LightGBM classification
lgb_cls = lgb.LGBMClassifier(n_estimators=200, num_leaves=31,
                              learning_rate=0.1, random_state=42)
lgb_cls.fit(X_tr, y_tr, eval_set=[(X_te, y_te)],
            callbacks=[lgb.early_stopping(20), lgb.log_evaluation(0)])

# CatBoost classification
cat_cls = CatBoostClassifier(iterations=200, depth=6,
                              learning_rate=0.1, random_seed=42, verbose=0)
cat_cls.fit(X_tr, y_tr, eval_set=(X_te, y_te), verbose=0)

for name, model in [('XGBoost', xgb_cls), ('LightGBM', lgb_cls),
                     ('CatBoost', cat_cls)]:
    pred = model.predict(X_te)
    acc = accuracy_score(y_te, pred)
    print(f"{name:<12} Accuracy: {acc:.4f}")

Feature Importance

All three libraries provide feature importance scores.

import matplotlib.pyplot as plt

# XGBoost
xgb.plot_importance(xgb_model, max_num_features=10)
plt.title("XGBoost Feature Importance")
plt.tight_layout()
plt.show()

# LightGBM
lgb.plot_importance(lgb_model, max_num_features=10)
plt.title("LightGBM Feature Importance")
plt.tight_layout()
plt.show()

# CatBoost
feature_imp = cat_model.get_feature_importance()
feature_names = housing.feature_names
sorted_idx = np.argsort(feature_imp)[-10:]
plt.barh(range(len(sorted_idx)),
         feature_imp[sorted_idx])
plt.yticks(range(len(sorted_idx)),
           [feature_names[i] for i in sorted_idx])
plt.title("CatBoost Feature Importance")
plt.tight_layout()
plt.show()

When to Choose Which

Choose LightGBM when you need speed and have large datasets with many features. Choose CatBoost when you have many categorical features and want minimal preprocessing. Choose XGBoost when you want the most mature ecosystem and widest community support.

Key Takeaways

Gradient boosting builds trees sequentially, each correcting the previous ensemble’s errors. XGBoost, LightGBM, and CatBoost all implement this idea with different optimizations. In practice, all three produce similar accuracy on most problems. The differences that matter are speed (LightGBM wins), categorical handling (CatBoost wins), and ecosystem maturity (XGBoost wins). Try all three with early stopping and pick the one that fits your workflow.