Model Evaluation Metrics: The Complete Guide

Picking the right evaluation metric is one of the most consequential decisions in a machine learning project. A model that maximizes the wrong metric can score well on a leaderboard while failing catastrophically in production. This guide covers every metric you will encounter in practice, with working code and clear guidance on when to reach for each one.

The Confusion Matrix

Every classification metric starts here. The confusion matrix is a 2x2 table (for binary classification) that counts how many predictions fell into each bucket.

                  Predicted Positive    Predicted Negative
Actual Positive        TP                     FN
Actual Negative        FP                     TN

TP = True Positive   (correctly predicted positive)
FP = False Positive  (predicted positive, actually negative)
FN = False Negative  (predicted negative, actually positive)
TN = True Negative   (correctly predicted negative)

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
import matplotlib.pyplot as plt

# Generate sample data
X, y = make_classification(n_samples=1000, n_features=20,
                           n_informative=10, weights=[0.7, 0.3],
                           random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                     test_size=0.2,
                                                     random_state=42)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Build and display the confusion matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)
# [[131   7]
#  [ 12  50]]

disp = ConfusionMatrixDisplay(confusion_matrix=cm,
                               display_labels=["Negative", "Positive"])
disp.plot(cmap="Blues")
plt.title("Confusion Matrix")
plt.show()

Reading the matrix: the diagonal contains correct predictions. Off-diagonal cells are errors. The top-right cell (FN) tells you how many positives you missed. The bottom-left cell (FP) tells you how many negatives you falsely flagged.

Accuracy

Accuracy is the fraction of predictions that are correct.

Accuracy = (TP + TN) / (TP + TN + FP + FN)

from sklearn.metrics import accuracy_score

acc = accuracy_score(y_test, y_pred)
print(f"Accuracy: {acc:.4f}")
# Accuracy: 0.9050

Accuracy works well when classes are balanced. On imbalanced datasets it becomes misleading. A spam filter that labels everything as “not spam” achieves 99% accuracy when only 1% of emails are spam, but it catches zero spam. Never use accuracy alone on imbalanced data.

Precision

Precision answers: of everything the model flagged as positive, how many actually were positive?

Precision = TP / (TP + FP)

from sklearn.metrics import precision_score

prec = precision_score(y_test, y_pred)
print(f"Precision: {prec:.4f}")
# Precision: 0.8772

High precision means few false alarms. Optimize for precision when the cost of a false positive is high, like flagging a legitimate transaction as fraud (which blocks the customer).

Recall (Sensitivity)

Recall answers: of all actual positives, how many did the model catch?

Recall = TP / (TP + FN)

from sklearn.metrics import recall_score

rec = recall_score(y_test, y_pred)
print(f"Recall: {rec:.4f}")
# Recall: 0.8065

High recall means few missed positives. Optimize for recall when missing a positive case is dangerous, like failing to detect cancer in a medical screening.

F1 Score

The F1 score is the harmonic mean of precision and recall. It balances both concerns into a single number.

F1 = 2 * (Precision * Recall) / (Precision + Recall)

from sklearn.metrics import f1_score

f1 = f1_score(y_test, y_pred)
print(f"F1 Score: {f1:.4f}")
# F1 Score: 0.8403

The harmonic mean punishes extreme imbalances. A model with 100% precision and 1% recall gets an F1 of 0.02, not 50.5%. Use F1 when you need a single metric that captures both precision and recall, especially on imbalanced datasets.

Macro, Micro, and Weighted F1

For multiclass problems, you need to decide how to aggregate per-class F1 scores.

from sklearn.metrics import f1_score, classification_report

# Multiclass example
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=42
)
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# Macro: unweighted mean of per-class F1
print(f"Macro F1: {f1_score(y_test, y_pred, average='macro'):.4f}")

# Weighted: weighted by class support
print(f"Weighted F1: {f1_score(y_test, y_pred, average='weighted'):.4f}")

# Full report
print(classification_report(y_test, y_pred, target_names=iris.target_names))

Use macro when all classes matter equally. Use weighted when you want to account for class imbalance.

AUC-ROC

The ROC curve plots the true positive rate (recall) against the false positive rate at every possible classification threshold. AUC-ROC is the area under that curve. It measures how well the model separates the two classes regardless of threshold choice.

TPR (Recall)
|        .---------
|      ./
|    ./
|  ./
| /
|/
+-------------------> FPR

AUC = 1.0  -> perfect separation
AUC = 0.5  -> random guessing (diagonal line)
AUC < 0.5  -> worse than random (flip predictions)

from sklearn.metrics import roc_auc_score, roc_curve
import matplotlib.pyplot as plt

# Binary classification example (using first dataset)
X, y = make_classification(n_samples=1000, n_features=20,
                           n_informative=10, weights=[0.7, 0.3],
                           random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                     test_size=0.2,
                                                     random_state=42)
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Get probability scores
y_proba = model.predict_proba(X_test)[:, 1]

# Calculate AUC
auc = roc_auc_score(y_test, y_proba)
print(f"AUC-ROC: {auc:.4f}")

# Plot ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_proba)
plt.plot(fpr, tpr, label=f"AUC = {auc:.3f}")
plt.plot([0, 1], [0, 1], "k--", label="Random")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
plt.title("ROC Curve")
plt.legend()
plt.show()

AUC-ROC is threshold-independent, which makes it useful for comparing models before you decide on a threshold. However, it can be overly optimistic on imbalanced datasets because the false positive rate denominator includes the large number of true negatives. In those cases, prefer AUC-PR (area under the precision-recall curve).

Regression Metrics

Classification metrics do not apply to regression problems. Here are the standard regression metrics.

Mean Absolute Error (MAE)

from sklearn.metrics import mean_absolute_error
import numpy as np

y_true = np.array([3.0, 5.0, 2.5, 7.0])
y_pred = np.array([2.8, 5.2, 2.0, 6.5])

mae = mean_absolute_error(y_true, y_pred)
print(f"MAE: {mae:.4f}")
# MAE: 0.3500

MAE measures average absolute deviation. It is robust to outliers and easy to interpret: “on average, predictions are off by X units.”

Mean Squared Error (MSE) and RMSE

from sklearn.metrics import mean_squared_error

mse = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)
print(f"MSE: {mse:.4f}")
print(f"RMSE: {rmse:.4f}")

MSE penalizes large errors quadratically. RMSE brings it back to the original units. Use MSE/RMSE when large errors are especially costly.

R-squared

from sklearn.metrics import r2_score

r2 = r2_score(y_true, y_pred)
print(f"R-squared: {r2:.4f}")

R-squared measures how much variance the model explains relative to a baseline that always predicts the mean. A value of 1.0 means perfect prediction, 0.0 means no better than the mean, and negative values mean worse than the mean.

Choosing the Right Metric

The right metric depends on what mistake costs more in your domain.

Putting It All Together

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score,
    f1_score, roc_auc_score, classification_report
)

def evaluate_classifier(y_true, y_pred, y_proba=None):
    """Print a comprehensive evaluation report."""
    print(f"Accuracy:  {accuracy_score(y_true, y_pred):.4f}")
    print(f"Precision: {precision_score(y_true, y_pred):.4f}")
    print(f"Recall:    {recall_score(y_true, y_pred):.4f}")
    print(f"F1 Score:  {f1_score(y_true, y_pred):.4f}")
    if y_proba is not None:
        print(f"AUC-ROC:   {roc_auc_score(y_true, y_proba):.4f}")
    print("\nClassification Report:")
    print(classification_report(y_true, y_pred))

# Use the function
evaluate_classifier(y_test, y_pred, y_proba)

Key Takeaways

Never default to accuracy without checking class balance. Always start with the confusion matrix to understand where errors cluster. Use F1 when precision and recall both matter. Use AUC-ROC to compare models before committing to a threshold. For regression, choose between MAE and RMSE based on how much you want to penalize large errors. The best metric is the one that aligns with the real-world cost of mistakes in your specific application.