Random Forests: A Practical Guide

A Random Forest is an ensemble of decision trees. Each tree is trained on a random subset of the data and features, and the forest combines their predictions through voting (classification) or averaging (regression). This reduces overfitting while keeping the power of decision trees.

How it works

1. Bootstrap sampling: draw N random samples (with replacement) from the training set
2. Feature randomness: at each split, consider only a random subset of features
3. Build tree: grow a full (or pruned) decision tree on the bootstrap sample
4. Repeat: build hundreds of independent trees
5. Aggregate: majority vote (classification) or average (regression)

Basic usage

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

print(f"Train accuracy: {rf.score(X_train, y_train):.3f}")
print(f"Test accuracy: {rf.score(X_test, y_test):.3f}")

Key hyperparameters

rf = RandomForestClassifier(
    n_estimators=200,        # number of trees (more = better, but slower)
    max_depth=10,            # maximum tree depth
    min_samples_split=5,     # minimum samples to split a node
    min_samples_leaf=2,      # minimum samples in a leaf
    max_features="sqrt",     # features to consider per split
    bootstrap=True,          # use bootstrap sampling
    oob_score=True,          # out-of-bag score (free validation)
    n_jobs=-1,               # use all CPU cores
    random_state=42,
)

Tuning with GridSearchCV

from sklearn.model_selection import GridSearchCV

param_grid = {
    "n_estimators": [100, 200, 500],
    "max_depth": [5, 10, 20, None],
    "min_samples_split": [2, 5, 10],
    "max_features": ["sqrt", "log2"],
}

grid = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
)
grid.fit(X_train, y_train)
print(f"Best params: {grid.best_params_}")
print(f"Best CV score: {grid.best_score_:.3f}")

Feature importance

import pandas as pd

importances = rf.feature_importances_
feature_names = load_breast_cancer().feature_names

importance_df = pd.DataFrame({
    "feature": feature_names,
    "importance": importances,
}).sort_values("importance", ascending=False)

print(importance_df.head(10))

Permutation importance

More reliable than built-in importance for correlated features:

from sklearn.inspection import permutation_importance

perm_imp = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)

for i in perm_imp.importances_mean.argsort()[::-1][:10]:
    print(f"{feature_names[i]}: {perm_imp.importances_mean[i]:.4f}")

Out-of-bag score

Each tree is trained on ~63% of the data (bootstrap). The remaining ~37% serves as a free validation set.

rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
rf.fit(X_train, y_train)
print(f"OOB score: {rf.oob_score_:.3f}")

Random Forest regression

from sklearn.ensemble import RandomForestRegressor

rf_reg = RandomForestRegressor(n_estimators=100, random_state=42)
rf_reg.fit(X_train, y_train)
predictions = rf_reg.predict(X_test)

When to use Random Forests

Use case	Good fit?
Tabular data with mixed features	Excellent
Quick baseline model	Excellent
Feature importance analysis	Good
Small datasets	Good
Real-time prediction (latency-sensitive)	Moderate (many trees = slower)
Image/text/sequence data	Poor (use deep learning)

Random Forest vs Gradient Boosting

Feature	Random Forest	Gradient Boosting
Training	Parallel (fast)	Sequential (slower)
Overfitting risk	Low	Higher
Tuning effort	Low	More hyperparameters
Raw accuracy	Good	Often slightly better
Interpretability	Moderate	Moderate

Summary

Random Forests combine hundreds of decision trees trained on random subsets of data and features. They are hard to overfit, require minimal tuning, and provide built-in feature importance. Use them as your first model for tabular data — they set a strong baseline with minimal effort.