Model Interpretability with SHAP Values

The Problem SHAP Solves

You train a model and it scores 94% accuracy. Your stakeholder asks: why did the model reject this particular application? You cannot answer that question with accuracy alone. You need a way to decompose a single prediction into the contribution of each feature.

SHAP (SHapley Additive exPlanations) does exactly this. It assigns each feature a value representing how much that feature pushed the prediction away from the average. SHAP values are grounded in cooperative game theory, which gives them strong theoretical guarantees that simpler methods like permutation importance lack.

The Intuition Behind Shapley Values

Imagine a prediction is made by a team of features working together. How do you fairly divide the credit? Shapley values, from game theory, answer this by considering every possible coalition of features.

For each feature, SHAP computes the average marginal contribution across all possible orderings of features. A feature that consistently improves predictions regardless of which other features are present gets a high SHAP value.

The key properties are:

• Efficiency: SHAP values for all features sum to the difference between the prediction and the average prediction.
• Symmetry: Two features that contribute equally get equal SHAP values.
• Null player: A feature that never changes the prediction gets a SHAP value of zero.

Installing SHAP

pip install shap

Computing SHAP Values

For Tree-Based Models

Tree-based models (XGBoost, LightGBM, random forests, gradient boosting) have a fast exact algorithm called TreeSHAP.

import shap
import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

# Train a model
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)
feature_names = data.feature_names

model = xgb.XGBClassifier(n_estimators=100, max_depth=4, random_state=42)
model.fit(X_train, y_train)

# Create SHAP explainer
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

print(f"SHAP values shape: {shap_values.shape}")
print(f"Base value (average prediction): {explainer.expected_value:.3f}")

The shap_values array has the same shape as X_test. Each entry tells you how much that feature pushed the prediction for that row.

For Any Model

For models without a specialized algorithm, use KernelSHAP. It is model-agnostic but slower.

from sklearn.neural_network import MLPClassifier

mlp = MLPClassifier(hidden_layer_sizes=(64, 32), max_iter=500, random_state=42)
mlp.fit(X_train, y_train)

# KernelSHAP needs a background dataset (a sample of training data)
background = shap.sample(X_train, 100)
explainer = shap.KernelExplainer(mlp.predict_proba, background)
shap_values = explainer.shap_values(X_test[:20])  # Slower, so use fewer samples

Explaining Individual Predictions

Waterfall Plot

A waterfall plot shows how each feature contributes to a single prediction, starting from the base value (average prediction) and ending at the model output.

import matplotlib.pyplot as plt

# Explain the first test sample
shap.waterfall_plot(
    shap.Explanation(
        values=shap_values[0],
        base_values=explainer.expected_value,
        data=X_test[0],
        feature_names=feature_names,
    )
)
plt.tight_layout()
plt.savefig("shap_waterfall.png", dpi=150)

Reading the plot: red arrows push the prediction higher, blue arrows push it lower. The length of each arrow is the magnitude of the SHAP value. Features are sorted by absolute SHAP value so the most important ones appear first.

Force Plot

A force plot is a compact alternative that shows the same information horizontally.

shap.force_plot(
    explainer.expected_value,
    shap_values[0],
    X_test[0],
    feature_names=feature_names,
    matplotlib=True,
)

Global Feature Importance

Beeswarm Plot

The beeswarm plot shows SHAP values for every sample in the dataset, giving you both importance and direction of effect.

shap.summary_plot(shap_values, X_test, feature_names=feature_names)

Each dot is a sample. The x-axis is the SHAP value. Color indicates the feature value (red for high, blue for low). A feature where high values (red dots) cluster on the right has a positive relationship with the prediction.

Bar Plot

For a simpler view, the bar plot shows mean absolute SHAP values.

shap.summary_plot(shap_values, X_test, feature_names=feature_names, plot_type="bar")

This is the SHAP equivalent of feature importance but more reliable because it accounts for feature interactions and is consistent across different models.

Dependence Plots

Dependence plots show how a single feature affects predictions across all samples. They reveal non-linear relationships and interactions.

# SHAP dependence plot for the most important feature
shap.dependence_plot(
    "worst radius",
    shap_values,
    X_test,
    feature_names=feature_names,
    interaction_index="worst concave points",
)

The x-axis is the feature value, the y-axis is the SHAP value. If the relationship is linear, you see a straight line. Curves indicate non-linear effects. Color from the interaction index reveals how a second feature modifies the relationship.

Using SHAP for Debugging

SHAP values can reveal problems that accuracy scores hide.

Detecting Data Leakage

If a feature that should not be available at prediction time has very high SHAP values, you likely have data leakage.

# Check for suspicious features
import numpy as np

mean_abs_shap = np.abs(shap_values).mean(axis=0)
top_features = sorted(
    zip(feature_names, mean_abs_shap),
    key=lambda x: x[1],
    reverse=True,
)

print("Top 5 features by mean |SHAP|:")
for name, value in top_features[:5]:
    print(f"  {name}: {value:.4f}")

If a feature like final_decision or outcome_date appears at the top, investigate immediately.

Finding Bias

SHAP can expose whether a model relies on sensitive features like gender or zip code. Even if you remove those features, the model might learn proxies. Check whether proxy features (like certain product categories or interaction patterns) have SHAP values that correlate with protected attributes.

import pandas as pd

# Check if SHAP values correlate with a sensitive attribute
shap_df = pd.DataFrame(shap_values, columns=feature_names)
correlation = shap_df.corrwith(pd.Series(X_test[:, sensitive_feature_idx]))
print(correlation.sort_values(ascending=False).head())

SHAP for Stakeholder Communication

Technical plots are great for data scientists. For business stakeholders, translate SHAP values into natural language.

def explain_prediction(shap_vals, feature_vals, feature_names, top_n=3):
    """Generate a plain-English explanation for a prediction."""
    pairs = sorted(
        zip(feature_names, shap_vals, feature_vals),
        key=lambda x: abs(x[1]),
        reverse=True,
    )

    reasons = []
    for name, shap_val, feat_val in pairs[:top_n]:
        direction = "increased" if shap_val > 0 else "decreased"
        reasons.append(f"{name}={feat_val:.2f} {direction} the score by {abs(shap_val):.3f}")

    return "; ".join(reasons)

explanation = explain_prediction(shap_values[0], X_test[0], feature_names)
print(f"This prediction was driven by: {explanation}")

This turns a SHAP explanation into a sentence like: “worst radius=17.99 increased the score by 2.341; worst concave points=0.27 increased the score by 1.876; mean concavity=0.30 increased the score by 0.943.”

Performance Considerations

TreeSHAP runs in polynomial time and handles thousands of samples quickly. KernelSHAP is exponential in theory but uses sampling to approximate. For large datasets, compute SHAP values on a representative sample rather than the full test set.

# For large datasets, sample before computing SHAP
sample_idx = np.random.choice(len(X_test), size=500, replace=False)
shap_values_sample = explainer.shap_values(X_test[sample_idx])

Summary

SHAP values decompose any prediction into per-feature contributions grounded in game theory. Use TreeSHAP for fast exact explanations of tree-based models and KernelSHAP for everything else. Waterfall plots explain individual predictions. Beeswarm plots reveal global patterns. Dependence plots expose non-linear relationships. Beyond interpretation, SHAP is a debugging tool that catches data leakage, proxy bias, and features that should not matter. When presenting to stakeholders, convert SHAP values into plain-language reasons.