Feature Engineering: Encoding, Scaling, Binning & Text

Feature engineering is the process of transforming raw data into representations that models can learn from effectively. It is often the single highest-leverage activity in a machine learning project. A well-engineered feature can do more for accuracy than switching algorithms. This guide covers the five core techniques: encoding, scaling, binning, feature crosses, and text features.

Encoding Categorical Variables

Categorical features like country, color, or product type need to be converted into numbers. The method you choose depends on the cardinality and the model.

Label Encoding

Assigns each category a unique integer. Works with tree-based models that can split on arbitrary thresholds. Bad for linear models because it introduces false ordinal relationships.

import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder

df = pd.DataFrame({
    'color': ['red', 'blue', 'green', 'red', 'blue', 'green'],
    'size': ['S', 'M', 'L', 'M', 'S', 'L'],
    'price': [10, 20, 30, 15, 25, 35]
})

le = LabelEncoder()
df['color_encoded'] = le.fit_transform(df['color'])
print(df[['color', 'color_encoded']])
# blue=0, green=1, red=2

One-Hot Encoding

Creates a binary column for each category. Safe for all model types but explodes feature count on high-cardinality columns.

from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
encoded = ohe.fit_transform(df[['color']])
feature_names = ohe.get_feature_names_out(['color'])

encoded_df = pd.DataFrame(encoded, columns=feature_names)
print(encoded_df)
# color_blue  color_green  color_red
# 0.0         0.0          1.0
# 1.0         0.0          0.0
# ...

Target Encoding

Replaces each category with the mean of the target variable for that category. Handles high cardinality well but risks target leakage without proper regularization.

from sklearn.model_selection import KFold
import numpy as np

def target_encode(df, col, target, n_splits=5):
    """Target encode with k-fold to prevent leakage."""
    encoded = pd.Series(np.nan, index=df.index)
    global_mean = df[target].mean()
    
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=42)
    for train_idx, val_idx in kf.split(df):
        means = df.iloc[train_idx].groupby(col)[target].mean()
        encoded.iloc[val_idx] = df.iloc[val_idx][col].map(means)
    
    # Fill any remaining NaN with global mean
    encoded = encoded.fillna(global_mean)
    return encoded

# Example with a larger dataset
np.random.seed(42)
data = pd.DataFrame({
    'city': np.random.choice(['NYC', 'LA', 'Chicago', 'Houston', 'Phoenix'], 1000),
    'target': np.random.randint(0, 2, 1000)
})
data['city_encoded'] = target_encode(data, 'city', 'target')
print(data.groupby('city')['city_encoded'].mean())

Frequency Encoding

Replaces categories with their frequency count. Simple, handles high cardinality, and does not leak target information.

freq_map = data['city'].value_counts(normalize=True)
data['city_freq'] = data['city'].map(freq_map)
print(data[['city', 'city_freq']].head(10))

Scaling Numerical Features

Standard Scaling (Z-score)

Centers features to mean=0, std=1. Essential for linear models, SVMs, and neural networks.

from sklearn.preprocessing import StandardScaler

X = np.array([[1, 1000], [2, 2000], [3, 3000], [4, 4000]])
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
print(f"Mean: {X_scaled.mean(axis=0)}")   # [0, 0]
print(f"Std:  {X_scaled.std(axis=0)}")    # [1, 1]

Min-Max Scaling

Scales features to a fixed range, typically [0, 1]. Useful when you need bounded values (e.g., for neural networks with sigmoid activations).

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(0, 1))
X_minmax = scaler.fit_transform(X)
print(X_minmax)
# [[0.   0.  ]
#  [0.33 0.33]
#  [0.67 0.67]
#  [1.   1.  ]]

Robust Scaling

Uses median and IQR instead of mean and std. Robust to outliers.

from sklearn.preprocessing import RobustScaler

X_with_outlier = np.array([[1, 100], [2, 200], [3, 300], [100, 10000]])
robust = RobustScaler()
X_robust = robust.fit_transform(X_with_outlier)
print(X_robust)
# Outlier row is less extreme compared to StandardScaler

When to Scale

Model Type	Needs Scaling?
Linear/logistic regression	Yes
SVM	Yes
KNN	Yes
Neural networks	Yes
Decision trees	No
Random forest / XGBoost	No

Binning (Discretization)

Binning converts continuous features into discrete buckets. It helps when the relationship between feature and target is non-linear and you are using a linear model.

from sklearn.preprocessing import KBinsDiscretizer

ages = np.array([22, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70]).reshape(-1, 1)

# Equal-width bins
binner = KBinsDiscretizer(n_bins=4, encode='ordinal', strategy='uniform')
age_binned = binner.fit_transform(ages)
print("Equal-width bins:")
print(np.column_stack([ages, age_binned]))

# Quantile bins (equal number of samples per bin)
binner_q = KBinsDiscretizer(n_bins=4, encode='ordinal', strategy='quantile')
age_binned_q = binner_q.fit_transform(ages)
print("\nQuantile bins:")
print(np.column_stack([ages, age_binned_q]))

Custom Bins Based on Domain Knowledge

# Custom age brackets
def age_bracket(age):
    if age < 25:
        return 'young'
    elif age < 40:
        return 'adult'
    elif age < 60:
        return 'middle_aged'
    else:
        return 'senior'

df_ages = pd.DataFrame({'age': [22, 35, 45, 67, 28, 55]})
df_ages['bracket'] = df_ages['age'].apply(age_bracket)
print(df_ages)

Feature Crosses

Feature crosses capture interactions between two or more features that the model might miss, especially linear models. A feature cross multiplies or concatenates features to create a new combined feature.

from sklearn.preprocessing import PolynomialFeatures

# Numerical feature crosses
X = pd.DataFrame({
    'bedrooms': [2, 3, 4, 2, 5],
    'sqft': [1000, 1500, 2000, 800, 3000],
    'age': [5, 10, 20, 2, 30]
})

# Create interaction features (degree=2, interaction_only=True)
poly = PolynomialFeatures(degree=2, interaction_only=True,
                           include_bias=False)
X_cross = poly.fit_transform(X)
feature_names = poly.get_feature_names_out(X.columns)
print("Crossed features:")
print(pd.DataFrame(X_cross, columns=feature_names).head())

Categorical Feature Crosses

# Cross two categorical features
df = pd.DataFrame({
    'city': ['NYC', 'LA', 'NYC', 'LA'],
    'device': ['mobile', 'desktop', 'desktop', 'mobile'],
    'clicked': [1, 0, 1, 0]
})

# Create cross feature
df['city_device'] = df['city'] + '_' + df['device']
print(df)

# Then one-hot encode the cross
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder(sparse_output=False)
cross_encoded = ohe.fit_transform(df[['city_device']])
print(pd.DataFrame(cross_encoded,
                    columns=ohe.get_feature_names_out()))

Ratio Features

df_house = pd.DataFrame({
    'price': [500000, 750000, 300000],
    'sqft': [2000, 3000, 1200],
    'bedrooms': [3, 4, 2]
})

df_house['price_per_sqft'] = df_house['price'] / df_house['sqft']
df_house['sqft_per_bedroom'] = df_house['sqft'] / df_house['bedrooms']
print(df_house)

Text Features

Bag of Words and TF-IDF

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

documents = [
    "machine learning is great",
    "deep learning is a subset of machine learning",
    "natural language processing uses machine learning",
    "NLP and deep learning are related"
]

# Bag of Words
bow = CountVectorizer()
X_bow = bow.fit_transform(documents)
print("BoW features:", bow.get_feature_names_out())
print("BoW matrix shape:", X_bow.shape)

# TF-IDF (better for most tasks)
tfidf = TfidfVectorizer(max_features=100, ngram_range=(1, 2))
X_tfidf = tfidf.fit_transform(documents)
print("\nTF-IDF features:", tfidf.get_feature_names_out()[:10])
print("TF-IDF matrix shape:", X_tfidf.shape)

Text Statistics as Features

def text_features(text):
    """Extract numerical features from text."""
    return {
        'char_count': len(text),
        'word_count': len(text.split()),
        'avg_word_length': np.mean([len(w) for w in text.split()]),
        'uppercase_ratio': sum(1 for c in text if c.isupper()) / len(text),
        'has_numbers': int(any(c.isdigit() for c in text)),
        'exclamation_count': text.count('!'),
        'question_count': text.count('?')
    }

sample_texts = [
    "Buy now! 50% OFF!!!",
    "Meeting scheduled for tomorrow at 3pm.",
    "URGENT: Your account has been compromised!"
]

text_df = pd.DataFrame([text_features(t) for t in sample_texts])
print(text_df)

Combining Text and Tabular Features

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

# Mixed dataset: text + numeric
data = pd.DataFrame({
    'description': ['cheap fast delivery', 'premium quality product',
                     'budget option good value', 'luxury brand item'],
    'price': [10, 100, 25, 200],
    'rating': [3.5, 4.8, 4.0, 4.5]
})

preprocessor = ColumnTransformer([
    ('text', TfidfVectorizer(max_features=50), 'description'),
    ('numeric', StandardScaler(), ['price', 'rating'])
])

X_combined = preprocessor.fit_transform(data)
print(f"Combined feature matrix shape: {X_combined.shape}")

Putting It Together: Full Pipeline

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import (
    StandardScaler, OneHotEncoder, FunctionTransformer
)
from sklearn.impute import SimpleImputer
from sklearn.ensemble import GradientBoostingClassifier

numeric_features = ['age', 'income']
categorical_features = ['city', 'device']

preprocessor = ColumnTransformer([
    ('num', Pipeline([
        ('imputer', SimpleImputer(strategy='median')),
        ('scaler', StandardScaler())
    ]), numeric_features),
    ('cat', Pipeline([
        ('imputer', SimpleImputer(strategy='most_frequent')),
        ('encoder', OneHotEncoder(handle_unknown='ignore',
                                   sparse_output=False))
    ]), categorical_features)
])

pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('classifier', GradientBoostingClassifier(random_state=42))
])

Key Takeaways

Use one-hot encoding for low-cardinality categoricals and target encoding for high-cardinality ones. Scale features before using distance-based models but skip scaling for tree-based models. Use binning when the feature-target relationship is non-linear. Create feature crosses to capture interactions that linear models cannot learn on their own. Extract TF-IDF features from text and combine them with numerical features in a ColumnTransformer. Always build these transformations inside a pipeline to prevent data leakage during cross-validation.