Feature Engineering Techniques: Encoding, Scaling, and Binning

Why Feature Engineering Matters

Raw data rarely arrives in a format that machine learning models can use directly. Feature engineering is the process of transforming raw inputs into representations that models understand. It is often the single highest-leverage activity in a machine learning project. A well-engineered feature can do more for accuracy than switching from logistic regression to a deep neural network.

There are three core categories of transformation that cover most tabular data work: encoding categorical variables, scaling numerical features, and binning continuous values. This article walks through each one with working Python code.

Encoding Categorical Variables

Categorical features like country, color, or product type need to be converted into numbers. The method you choose depends on the cardinality of the feature and the model you plan to use.

Label Encoding

Label encoding assigns each category a unique integer. It works well with tree-based models like random forests and gradient boosting, which can split on arbitrary thresholds. It is a poor choice for linear models because it introduces a false ordinal relationship.

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
df["color_encoded"] = le.fit_transform(df["color"])

# color: ["red", "blue", "green"] -> [2, 0, 1]

One-Hot Encoding

One-hot encoding creates a binary column for each category. It avoids the ordinal problem but can explode the feature space when cardinality is high.

from sklearn.preprocessing import OneHotEncoder
import pandas as pd

ohe = OneHotEncoder(sparse_output=False, drop="first")
encoded = ohe.fit_transform(df[["color"]])
encoded_df = pd.DataFrame(encoded, columns=ohe.get_feature_names_out())

df = pd.concat([df, encoded_df], axis=1)

The drop="first" parameter removes one column to avoid multicollinearity in linear models. If you have a feature with 200 categories, one-hot encoding produces 199 columns. In that case, consider target encoding instead.

Target Encoding

Target encoding replaces each category with the mean of the target variable for that category. It handles high-cardinality features elegantly but introduces a risk of target leakage. Always compute the encoding on training data only and apply smoothing.

from sklearn.preprocessing import TargetEncoder

te = TargetEncoder(smooth="auto")
df["city_encoded"] = te.fit_transform(df[["city"]], df["target"])

The smooth parameter blends the category mean with the global mean to reduce overfitting on rare categories. With five rows for a city, you get heavy regularization toward the global mean. With five thousand rows, the category mean dominates.

Scaling Numerical Features

Numerical features often have wildly different ranges. A salary column spans tens of thousands while an age column spans tens. Models that compute distances (KNN, SVM) or use gradient descent (linear regression, neural networks) are sensitive to scale. Tree-based models are not, but scaling rarely hurts.

Standard Scaling (Z-Score Normalization)

Standard scaling subtracts the mean and divides by the standard deviation. Each feature ends up with a mean of zero and a standard deviation of one.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df[["age", "salary"]] = scaler.fit_transform(df[["age", "salary"]])

Use standard scaling when your features are roughly normally distributed. It is the default choice for most linear models and neural networks.

Min-Max Scaling

Min-max scaling rescales features to a fixed range, typically zero to one. It preserves the shape of the original distribution but is sensitive to outliers.

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(0, 1))
df[["age", "salary"]] = scaler.fit_transform(df[["age", "salary"]])

A single extreme outlier in the salary column can compress all other values into a narrow band near zero. If outliers are present, prefer robust scaling.

Robust Scaling

Robust scaling uses the median and interquartile range instead of the mean and standard deviation. It is far less affected by outliers.

from sklearn.preprocessing import RobustScaler

scaler = RobustScaler()
df[["salary"]] = scaler.fit_transform(df[["salary"]])

This is the best default when you suspect outliers but do not want to remove them.

Binning Continuous Features

Binning converts a continuous feature into a set of discrete buckets. A raw age of 34 becomes “30-39”. This can help linear models capture non-linear relationships and reduce the impact of minor noise.

Equal-Width Binning

Equal-width binning divides the range into bins of the same size. It is simple but can produce bins with very few observations if the distribution is skewed.

df["age_bin"] = pd.cut(df["age"], bins=5, labels=False)

Quantile Binning

Quantile binning creates bins with roughly equal numbers of observations. This is more robust to skewed distributions.

df["salary_bin"] = pd.qcut(df["salary"], q=4, labels=["Q1", "Q2", "Q3", "Q4"])

Custom Binning with Domain Knowledge

Often the best bins come from domain knowledge rather than statistics. For a credit scoring model, age brackets like 18-25, 26-35, 36-50, and 50+ might align with known risk profiles.

bins = [0, 25, 35, 50, 100]
labels = ["young", "early_career", "mid_career", "senior"]
df["age_group"] = pd.cut(df["age"], bins=bins, labels=labels)

Putting It All Together in a Pipeline

Scikit-learn pipelines let you chain transformations and ensure that fitting happens only on training data. This prevents data leakage.

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier

numeric_features = ["age", "salary", "years_experience"]
categorical_features = ["department", "education_level"]

preprocessor = ColumnTransformer(
    transformers=[
        ("num", StandardScaler(), numeric_features),
        ("cat", OneHotEncoder(drop="first", handle_unknown="ignore"), categorical_features),
    ]
)

pipeline = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", RandomForestClassifier(n_estimators=200, random_state=42)),
])

pipeline.fit(X_train, y_train)
score = pipeline.score(X_test, y_test)
print(f"Accuracy: {score:.3f}")

The ColumnTransformer applies different transformations to different columns. The pipeline ensures that fit is called only on training data and transform is applied consistently to both training and test data.

Common Pitfalls

Fitting scalers or encoders on the full dataset before splitting introduces data leakage. Always fit on training data and transform test data with the same fitted object.

Over-binning destroys useful information. Start with few bins and increase only if validation performance improves.

One-hot encoding high-cardinality features creates sparse, noisy matrices. Switch to target encoding or embeddings when categories exceed a few dozen.

Dropping the original feature after encoding is easy to forget. Leaving both the raw and encoded versions in the dataset can confuse the model or slow training without benefit.

Summary

Feature engineering is not glamorous, but it is where most practical ML performance gains come from. Encoding converts categories to numbers. Scaling puts features on equal footing. Binning introduces non-linearity for models that need help. Wrap everything in a pipeline, fit only on training data, and validate that each transformation actually improves your metric before keeping it.