Data Preprocessing Pipeline: Missing Data, Outliers, and Imbalanced Classes

Real-world data is messy. Columns have missing values, outliers distort distributions, and class labels are rarely balanced. Skipping preprocessing or doing it incorrectly is the most common source of poor model performance and subtle data leakage. This guide shows you how to handle each problem properly, wired together in a reusable pipeline.

Handling Missing Data

Detecting Missing Values

import pandas as pd
import numpy as np

# Simulate messy data
np.random.seed(42)
n = 1000
df = pd.DataFrame({
    'age': np.random.randint(18, 80, n).astype(float),
    'income': np.random.lognormal(10, 1, n),
    'education': np.random.choice(['high_school', 'bachelor', 'master', 'phd', None], n),
    'score': np.random.randn(n) * 10 + 50,
    'target': np.random.randint(0, 2, n)
})

# Inject missing values
df.loc[np.random.choice(n, 50, replace=False), 'age'] = np.nan
df.loc[np.random.choice(n, 80, replace=False), 'income'] = np.nan
df.loc[np.random.choice(n, 30, replace=False), 'score'] = np.nan

# Check missing patterns
print(df.isnull().sum())
print(f"\nTotal missing: {df.isnull().sum().sum()}")
print(f"Rows with any missing: {df.isnull().any(axis=1).sum()}")

Imputation Strategies

from sklearn.impute import SimpleImputer, KNNImputer

# Mean imputation -- fast, works for MCAR data
mean_imputer = SimpleImputer(strategy='mean')

# Median imputation -- robust to outliers
median_imputer = SimpleImputer(strategy='median')

# Mode imputation -- for categorical features
mode_imputer = SimpleImputer(strategy='most_frequent')

# KNN imputation -- uses similar rows to fill gaps
knn_imputer = KNNImputer(n_neighbors=5)

# Example: apply KNN imputer to numeric columns
numeric_cols = ['age', 'income', 'score']
df[numeric_cols] = knn_imputer.fit_transform(df[numeric_cols])

print(f"Missing after imputation: {df[numeric_cols].isnull().sum().sum()}")

Adding Missing Indicators

Sometimes the fact that a value is missing is informative. Add a binary column to capture this signal.

from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

# Create missing indicator + imputer together
imputer_with_indicator = SimpleImputer(strategy='median',
                                        add_indicator=True)

# This outputs the imputed values followed by binary missing flags
result = imputer_with_indicator.fit_transform(df[['age', 'income', 'score']])
print(f"Shape: {result.shape}")  # (1000, 6) -- 3 values + 3 indicators

Detecting and Treating Outliers

Statistical Methods

# Z-score method
from scipy import stats

z_scores = np.abs(stats.zscore(df['income'].dropna()))
outlier_mask_z = z_scores > 3
print(f"Z-score outliers: {outlier_mask_z.sum()}")

# IQR method -- more robust
Q1 = df['income'].quantile(0.25)
Q3 = df['income'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outlier_mask_iqr = (df['income'] < lower) | (df['income'] > upper)
print(f"IQR outliers: {outlier_mask_iqr.sum()}")
print(f"Bounds: [{lower:.0f}, {upper:.0f}]")

Treatment Strategies

# Strategy 1: Cap (winsorize) at percentiles
def winsorize_column(series, lower_pct=0.01, upper_pct=0.99):
    lower = series.quantile(lower_pct)
    upper = series.quantile(upper_pct)
    return series.clip(lower, upper)

df['income_capped'] = winsorize_column(df['income'])

# Strategy 2: Log transform to reduce skew
df['income_log'] = np.log1p(df['income'])

# Strategy 3: Isolation Forest for multivariate outlier detection
from sklearn.ensemble import IsolationForest

iso_forest = IsolationForest(contamination=0.05, random_state=42)
outlier_labels = iso_forest.fit_predict(df[['age', 'income', 'score']])
print(f"Isolation Forest outliers: {(outlier_labels == -1).sum()}")

# Remove outliers
df_clean = df[outlier_labels == 1].copy()
print(f"Rows after removal: {len(df_clean)}")

When to Remove vs Cap vs Transform

• Remove when outliers are clearly data errors (negative ages, impossible values).
• Cap when outliers are real but you want to limit their influence.
• Log transform when the distribution is right-skewed (common for monetary values).
• Keep when using tree-based models, which are naturally robust to outliers.

Handling Imbalanced Classes

Understanding the Problem

from sklearn.datasets import make_classification

X, y = make_classification(n_samples=10000, n_features=20,
                           n_informative=10, weights=[0.95, 0.05],
                           random_state=42)

print(f"Class distribution: {np.bincount(y)}")
print(f"Positive rate: {y.mean():.3f}")
# Class 0: ~9500, Class 1: ~500

A model that always predicts the majority class gets 95% accuracy but catches zero minority cases.

Resampling Techniques

from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler
from imblearn.combine import SMOTETomek

# Random oversampling -- duplicate minority samples
ros = RandomOverSampler(random_state=42)
X_ros, y_ros = ros.fit_resample(X, y)
print(f"After oversampling: {np.bincount(y_ros)}")

# SMOTE -- synthesize new minority samples
smote = SMOTE(random_state=42)
X_smote, y_smote = smote.fit_resample(X, y)
print(f"After SMOTE: {np.bincount(y_smote)}")

# Random undersampling -- drop majority samples
rus = RandomUnderSampler(random_state=42)
X_rus, y_rus = rus.fit_resample(X, y)
print(f"After undersampling: {np.bincount(y_rus)}")

# Combined: SMOTE + Tomek link cleaning
smt = SMOTETomek(random_state=42)
X_smt, y_smt = smt.fit_resample(X, y)
print(f"After SMOTETomek: {np.bincount(y_smt)}")

Class Weights

Most classifiers support class_weight='balanced', which adjusts the loss function to penalize minority-class errors more heavily. This is simpler and often as effective as resampling.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

# Without class weights
model_default = RandomForestClassifier(n_estimators=100, random_state=42)
scores_default = cross_val_score(model_default, X, y, cv=5, scoring='f1')
print(f"Default F1: {scores_default.mean():.4f}")

# With class weights
model_balanced = RandomForestClassifier(n_estimators=100,
                                         class_weight='balanced',
                                         random_state=42)
scores_balanced = cross_val_score(model_balanced, X, y, cv=5, scoring='f1')
print(f"Balanced F1: {scores_balanced.mean():.4f}")

Threshold Tuning

Instead of using the default 0.5 threshold, tune it to maximize your target metric.

from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score, precision_recall_curve

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2,
                                                      random_state=42)
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Get probabilities
y_proba = model.predict_proba(X_test)[:, 1]

# Find optimal threshold
precisions, recalls, thresholds = precision_recall_curve(y_test, y_proba)
f1_scores = 2 * (precisions * recalls) / (precisions + recalls + 1e-8)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]

print(f"Default threshold (0.5) F1: {f1_score(y_test, y_proba >= 0.5):.4f}")
print(f"Optimal threshold ({best_threshold:.3f}) F1: {f1_scores[best_idx]:.4f}")

Building the Full Pipeline

Wire everything together into a single, leak-free pipeline.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

# Define column groups
numeric_features = ['age', 'income', 'score']
categorical_features = ['education']

# Numeric pipeline: impute -> scale
numeric_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='median', add_indicator=True)),
    ('scaler', StandardScaler())
])

# Categorical pipeline: impute -> encode
categorical_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('encoder', OneHotEncoder(handle_unknown='ignore', sparse_output=False))
])

# Combine into a single preprocessor
preprocessor = ColumnTransformer([
    ('num', numeric_pipeline, numeric_features),
    ('cat', categorical_pipeline, categorical_features)
])

# Full pipeline: preprocess -> classify
full_pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', GradientBoostingClassifier(
        n_estimators=200, max_depth=5, random_state=42
    ))
])

# Evaluate with cross-validation -- no leakage!
X_features = df[numeric_features + categorical_features]
y_target = df['target']

scores = cross_val_score(full_pipeline, X_features, y_target,
                         cv=5, scoring='f1')
print(f"Pipeline CV F1: {scores.mean():.4f} (+/- {scores.std():.4f})")

Raw Data
 |
 v
[Split into train/test] <-- FIRST step, prevents leakage
 |
 v
[ColumnTransformer]
 |-- numeric: Impute (median) -> Add missing flags -> Scale
 |-- categorical: Impute (mode) -> One-hot encode
 |
 v
[Classifier]
 |
 v
Predictions

Common Mistakes

1. Fitting imputers/scalers on the full dataset before splitting. This leaks test-set statistics into training. Always fit on training data only. Pipelines handle this automatically.

2. Applying SMOTE before cross-validation. Synthetic samples from the validation set leak into training. Use imblearn.pipeline.Pipeline to apply SMOTE inside the CV loop.

3. Removing outliers from the test set. The test set should represent real-world data, outliers included. Only clean the training set.

4. Using mean imputation for skewed features. The mean is pulled by outliers. Use median for skewed distributions.

Key Takeaways

Handle missing values with imputation (median for numeric, mode for categorical) and add missing indicators when missingness is informative. Treat outliers with capping or log transforms, but leave them alone for tree-based models. Address class imbalance with class weights first (simplest), SMOTE second, and threshold tuning always. Wire everything into a scikit-learn Pipeline to prevent data leakage. The pipeline ensures that all transformations are fit only on training data, even inside cross-validation loops.