Gradient Descent: Intuition and Implementation

Gradient descent is how machine learning models learn. It finds the parameters that minimize a cost function by iteratively taking small steps in the direction of steepest descent.

The intuition

Imagine you are blindfolded on a hilly landscape. You want to reach the lowest point. Your strategy: feel the slope under your feet and take a step downhill. Repeat until the ground feels flat.

That is gradient descent. The “slope” is the gradient (partial derivatives), and the “step size” is the learning rate.

The algorithm

# Pseudocode
repeat until convergence:
    gradient = compute_gradient(parameters, data)
    parameters = parameters - learning_rate * gradient

In code:

import numpy as np

def gradient_descent(X, y, lr=0.01, epochs=1000):
    m, n = X.shape
    w = np.zeros(n)
    b = 0.0

    for _ in range(epochs):
        predictions = X @ w + b
        errors = predictions - y

        dw = (1 / m) * (X.T @ errors)
        db = (1 / m) * np.sum(errors)

        w -= lr * dw
        b -= lr * db

    return w, b

The learning rate

The learning rate controls step size.

• Too small: converges very slowly, may take millions of steps
• Too large: overshoots the minimum, may diverge
• Just right: converges efficiently in hundreds or thousands of steps

# Visualizing different learning rates
for lr in [0.001, 0.01, 0.1, 1.0]:
    w, b = gradient_descent(X, y, lr=lr, epochs=100)
    print(f"lr={lr}: w={w[0]:.4f}, cost={cost(X, y, w, b):.4f}")

Batch gradient descent

Uses the entire dataset to compute the gradient at each step.

# Batch GD
gradient = (1 / m) * X.T @ (X @ w - y)
w -= lr * gradient

Pros: stable convergence, smooth cost curve. Cons: slow for large datasets (computes gradient over all samples every step).

Stochastic gradient descent (SGD)

Uses one random sample per step.

for i in np.random.permutation(m):
    xi = X[i:i+1]
    yi = y[i:i+1]
    gradient = xi.T @ (xi @ w - yi)
    w -= lr * gradient

Pros: fast updates, can escape local minima. Cons: noisy, erratic cost curve.

Mini-batch gradient descent

Uses a small batch (32-256 samples) per step. The standard in practice.

batch_size = 32
for start in range(0, m, batch_size):
    end = start + batch_size
    X_batch = X[start:end]
    y_batch = y[start:end]

    gradient = (1 / len(y_batch)) * X_batch.T @ (X_batch @ w - y_batch)
    w -= lr * gradient

Best of both worlds: faster than batch GD, smoother than SGD.

Momentum

SGD oscillates. Momentum dampens oscillations by accumulating a “velocity.”

velocity = np.zeros_like(w)
beta = 0.9

for epoch in range(epochs):
    gradient = compute_gradient(w)
    velocity = beta * velocity + (1 - beta) * gradient
    w -= lr * velocity

Think of a ball rolling downhill — momentum keeps it moving through small bumps.

Adam optimizer

Adam combines momentum with adaptive learning rates. It is the default optimizer for deep learning.

m_w = np.zeros_like(w)  # first moment
v_w = np.zeros_like(w)  # second moment
beta1, beta2 = 0.9, 0.999
epsilon = 1e-8

for t in range(1, epochs + 1):
    gradient = compute_gradient(w)

    m_w = beta1 * m_w + (1 - beta1) * gradient
    v_w = beta2 * v_w + (1 - beta2) * gradient ** 2

    m_hat = m_w / (1 - beta1 ** t)
    v_hat = v_w / (1 - beta2 ** t)

    w -= lr * m_hat / (np.sqrt(v_hat) + epsilon)

Comparison

Method	Speed	Stability	Memory
Batch GD	Slow	Very stable	High
SGD	Fast per step	Noisy	Low
Mini-batch	Fast	Balanced	Moderate
SGD + Momentum	Fast	Good	Low
Adam	Fast	Excellent	Moderate

Learning rate schedules

Reduce the learning rate over time for better convergence.

def lr_schedule(epoch, initial_lr=0.1):
    return initial_lr / (1 + 0.01 * epoch)

Summary

Gradient descent is the core optimization algorithm in machine learning. Use mini-batch SGD for training at scale. Add momentum to smooth oscillations. Use Adam for a robust default that adapts learning rates per parameter. The learning rate is the most important hyperparameter — start with 0.001 for Adam and tune from there.