Dimensionality Reduction: PCA, t-SNE, UMAP, and Feature Selection

High-dimensional data is hard to visualize, slow to process, and often contains redundant or noisy features. Dimensionality reduction compresses data into fewer dimensions while preserving the signal that matters. This guide covers the four main approaches: linear projection (PCA), nonlinear embedding (t-SNE, UMAP), and feature selection.

PCA: Principal Component Analysis

PCA finds new orthogonal axes (principal components) ordered by how much variance they capture. The first component points in the direction of maximum spread, the second in the next best perpendicular direction, and so on.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler

# Load handwritten digits dataset (64 features, 10 classes)
digits = load_digits()
X, y = digits.data, digits.target

# Always scale before PCA
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Fit PCA
pca = PCA()
X_pca = pca.fit_transform(X_scaled)

# Variance explained
cumulative_var = np.cumsum(pca.explained_variance_ratio_)
n_95 = np.argmax(cumulative_var >= 0.95) + 1
print(f"Components for 95% variance: {n_95} (out of {X.shape[1]})")

# Plot cumulative variance
plt.plot(cumulative_var)
plt.axhline(y=0.95, color='r', linestyle='--', label='95% threshold')
plt.axvline(x=n_95, color='g', linestyle='--', label=f'n={n_95}')
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance')
plt.title('PCA Variance Explained')
plt.legend()
plt.show()

Choosing the Number of Components

# Method 1: Keep 95% of variance
pca_95 = PCA(n_components=0.95)
X_reduced = pca_95.fit_transform(X_scaled)
print(f"Reduced shape: {X_reduced.shape}")

# Method 2: Fixed number for visualization
pca_2d = PCA(n_components=2)
X_2d = pca_2d.fit_transform(X_scaled)

plt.scatter(X_2d[:, 0], X_2d[:, 1], c=y, cmap='tab10', s=5, alpha=0.7)
plt.colorbar(label='Digit')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.title('PCA 2D Projection of Digits')
plt.show()

When PCA Helps and Hurts

PCA works well when data lies on a linear subspace. It fails when the important structure is nonlinear (e.g., a Swiss roll). It also destroys interpretability since principal components are linear combinations of all original features.

t-SNE: t-Distributed Stochastic Neighbor Embedding

t-SNE is a nonlinear technique designed for visualization. It preserves local structure, meaning nearby points in high dimensions stay nearby in the embedding.

from sklearn.manifold import TSNE

# t-SNE is slow on large datasets -- subsample if needed
X_subset = X_scaled[:1500]
y_subset = y[:1500]

tsne = TSNE(n_components=2, perplexity=30, random_state=42,
            n_iter=1000, learning_rate='auto', init='pca')
X_tsne = tsne.fit_transform(X_subset)

plt.figure(figsize=(10, 8))
scatter = plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y_subset,
                       cmap='tab10', s=10, alpha=0.7)
plt.colorbar(scatter, label='Digit')
plt.title('t-SNE Visualization of Digits')
plt.show()

Key Parameters

• perplexity (5-50): Controls the balance between local and global structure. Lower values focus on local neighborhoods. Try 20-50 for most datasets.
• n_iter: Number of optimization iterations. Use at least 1000.
• learning_rate: 'auto' works well in modern scikit-learn.

t-SNE Pitfalls

1. Cluster sizes are meaningless. t-SNE adjusts density, so large clusters in the plot are not necessarily larger in the data.
2. Distances between clusters are meaningless. Two clusters far apart in the plot might be close in the original space.
3. Different runs give different results. Use random_state for reproducibility.
4. It does not support transforming new data. You cannot call transform() on unseen points.

UMAP: Uniform Manifold Approximation and Projection

UMAP is faster than t-SNE, better preserves global structure, and supports transforming new data. It has become the default for visualization and is also useful as a preprocessing step.

import umap

# UMAP for visualization
reducer = umap.UMAP(n_components=2, n_neighbors=15,
                     min_dist=0.1, random_state=42)
X_umap = reducer.fit_transform(X_scaled)

plt.figure(figsize=(10, 8))
scatter = plt.scatter(X_umap[:, 0], X_umap[:, 1], c=y,
                       cmap='tab10', s=5, alpha=0.7)
plt.colorbar(scatter, label='Digit')
plt.title('UMAP Visualization of Digits')
plt.show()

Key Parameters

• n_neighbors (5-50): Controls local vs global balance. Higher values preserve more global structure.
• min_dist (0.0-1.0): Controls how tightly points cluster. Lower values create denser clusters.
• n_components: Output dimensions. 2 for visualization, higher for preprocessing.

UMAP for Preprocessing

Unlike t-SNE, UMAP can be used as a feature extraction step before classification.

from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

# UMAP as preprocessing
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('umap', umap.UMAP(n_components=10, n_neighbors=15, random_state=42)),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

scores = cross_val_score(pipeline, X, y, cv=5, scoring='accuracy')
print(f"UMAP + RF accuracy: {scores.mean():.4f}")

# Compare with PCA preprocessing
pipeline_pca = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA(n_components=10)),
    ('classifier', RandomForestClassifier(n_estimators=100, random_state=42))
])

scores_pca = cross_val_score(pipeline_pca, X, y, cv=5, scoring='accuracy')
print(f"PCA + RF accuracy:  {scores_pca.mean():.4f}")

Feature Selection

Instead of creating new dimensions, feature selection picks a subset of original features. This preserves interpretability.

Filter Methods

Score each feature independently using statistical tests.

from sklearn.feature_selection import (
    SelectKBest, f_classif, mutual_info_classif
)

# ANOVA F-test
selector_f = SelectKBest(f_classif, k=20)
X_f = selector_f.fit_transform(X_scaled, y)
print(f"F-test selected shape: {X_f.shape}")

# Mutual information
selector_mi = SelectKBest(mutual_info_classif, k=20)
X_mi = selector_mi.fit_transform(X_scaled, y)
print(f"MI selected shape: {X_mi.shape}")

# See which features were selected
selected_mask = selector_f.get_support()
selected_indices = np.where(selected_mask)[0]
print(f"Selected feature indices: {selected_indices}")

Wrapper Methods: Recursive Feature Elimination

Train a model, remove the least important feature, and repeat.

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, random_state=42)
rfe = RFE(model, n_features_to_select=20, step=5)
rfe.fit(X_scaled, y)

print(f"Selected features: {np.where(rfe.support_)[0]}")
print(f"Feature ranking: {rfe.ranking_}")

# Evaluate with selected features
X_rfe = rfe.transform(X_scaled)
scores = cross_val_score(model, X_rfe, y, cv=5, scoring='accuracy')
print(f"RFE accuracy: {scores.mean():.4f}")

Embedded Methods: L1 Regularization

L1 (Lasso) regularization drives unimportant feature weights to exactly zero.

from sklearn.linear_model import LogisticRegression
from sklearn.feature_selection import SelectFromModel

# L1 regularization selects features automatically
l1_model = LogisticRegression(penalty='l1', solver='saga',
                               C=0.1, max_iter=5000, random_state=42)
l1_model.fit(X_scaled, y)

# Count non-zero features per class
n_nonzero = (l1_model.coef_ != 0).any(axis=0).sum()
print(f"Features with non-zero weight: {n_nonzero} / {X.shape[1]}")

# Use SelectFromModel for convenience
selector = SelectFromModel(l1_model, prefit=True)
X_l1 = selector.transform(X_scaled)
print(f"L1 selected shape: {X_l1.shape}")

Tree-Based Feature Importance

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=200, random_state=42)
model.fit(X_scaled, y)

importances = model.feature_importances_
top_k = 20
top_indices = np.argsort(importances)[-top_k:]

plt.barh(range(top_k), importances[top_indices])
plt.yticks(range(top_k), [f"Feature {i}" for i in top_indices])
plt.xlabel('Importance')
plt.title(f'Top {top_k} Features by RF Importance')
plt.tight_layout()
plt.show()

Comparison Table

Method	Type	Preserves	Speed	New Data	Use Case
PCA	Linear	Global variance	Fast	Yes	Preprocessing, compression
t-SNE	Nonlinear	Local structure	Slow	No	Visualization only
UMAP	Nonlinear	Local + global	Fast	Yes	Visualization, preprocessing
Filter	Selection	Interpretability	Fast	Yes	Quick screening
RFE	Selection	Interpretability	Slow	Yes	Careful feature selection
L1/Lasso	Selection	Interpretability	Medium	Yes	Sparse models

Key Takeaways

Use PCA when you need a fast, linear reduction that preserves variance. Use UMAP for visualization and nonlinear preprocessing. Use t-SNE only for visualization and be skeptical of cluster sizes and distances. Use feature selection when interpretability matters. Always scale your data before PCA and t-SNE. For preprocessing pipelines, put the dimensionality reduction step inside the pipeline to prevent data leakage during cross-validation.