System Design: Instagram Architecture Deep Dive

Instagram launched in October 2010. Within two hours, the servers crashed. Within 24 hours, 25,000 people had signed up. Within two months, one million users. By the time Facebook acquired it in 2012 for $1 billion, Instagram had 30 million users and just 13 employees. Today, it serves over 2 billion monthly active users.

What makes Instagram’s architecture remarkable is not exotic technology — it is how far they pushed ordinary tools. The backend still runs on Python/Django. The primary database is still PostgreSQL. The lesson is that boring technology, applied thoughtfully, scales further than most people assume.

The Feed: Push vs Pull

Instagram’s core experience is the feed — a personalized stream of posts from people you follow. Generating this feed at scale is one of the hardest problems in social media architecture.

The Fundamental Tradeoff

There are two approaches to feed generation:

Fanout on Write (Push Model): When a user posts a photo, immediately write a reference to that post into the feed of every follower. When a follower opens the app, their feed is pre-computed — just read it.

Fanout on Read (Pull Model): Store nothing pre-computed. When a user opens the app, query all the people they follow, fetch their recent posts, merge and rank them in real-time.

Fanout on Write:
  User posts photo
    -> For each follower, insert post_id into their feed table
    -> 1000 followers = 1000 writes
    -> Opening feed = 1 read (pre-computed)

Fanout on Read:
  User opens feed
    -> Query: "Who do I follow?" -> 500 users
    -> For each: "What did they post recently?" -> 500 queries
    -> Merge, rank, return top 50
    -> Opening feed = 500+ reads (computed on demand)

Instagram uses primarily fanout on write. When someone posts, the system pushes a reference to that post into a feed cache for each of their followers. This makes reads fast — opening Instagram requires reading from a single pre-computed list rather than aggregating across hundreds of sources.

The Celebrity Problem

Fanout on write has a weakness: users with millions of followers. When Cristiano Ronaldo (600M+ followers) posts a photo, writing to 600 million feed caches would take enormous time and resources.

Instagram handles this with a hybrid approach:

• Normal users (under ~10,000 followers): full fanout on write
• Celebrity accounts: their posts are NOT fanned out. Instead, when you open your feed, the system merges your pre-computed feed with a real-time pull of recent posts from celebrities you follow

Hybrid Feed Generation:

1. Pre-computed feed (Redis)
   -> Contains post references from non-celebrity follows
   -> Written to when those users post

2. Celebrity post pull (at read time)
   -> User follows 5 celebrity accounts
   -> Fetch their last 24h of posts (5 queries)
   -> Merge with pre-computed feed
   -> Rank by ML model (engagement prediction)
   -> Return top posts

This avoids writing to 600M feeds while keeping
read latency low (only a handful of celebrity queries).

Feed Ranking

Instagram shifted from chronological to ranked feeds in 2016. The ranking model considers:

• Relationship signals — do you frequently like or comment on this person’s posts?
• Interest signals — does this post match content types you engage with?
• Timeliness — newer posts get a boost, but a high-relevance post from 8 hours ago beats a low-relevance post from 1 hour ago
• Usage patterns — if you check Instagram every 30 minutes, show only the best posts since your last visit. If you check once a day, show a wider selection.

Image Processing Pipeline

Instagram processes hundreds of millions of photo and video uploads per day. Each upload triggers a pipeline that transforms the raw media into multiple formats optimized for different devices and contexts.

Upload Flow

Image Upload Pipeline:

1. Client Upload
   -> Image uploaded to edge server (CDN PoP)
   -> Generates unique media ID
   -> Returns ID to client immediately (async processing begins)

2. Processing Queue (RabbitMQ/Celery)
   -> Original stored in S3
   -> Tasks queued:
      a. Generate multiple resolutions:
         - 1080x1080 (feed)
         - 640x640 (feed, older devices)
         - 320x320 (thumbnails)
         - 150x150 (profile grid)
      b. Apply filter (if selected)
      c. Run content moderation (ML classifier)
      d. Extract EXIF data (location, camera)
      e. Generate blurhash placeholder

3. Storage
   -> Processed images stored in S3
   -> Metadata written to PostgreSQL
   -> CDN URLs generated for each resolution

4. Feed Fanout
   -> Post reference pushed to followers' feeds
   -> Push notification sent (if enabled)

Storage Scale

Instagram was one of the earliest large-scale users of Amazon S3. The volume is staggering — with over 100 million photos uploaded daily, and each photo stored in 4-5 resolutions, that is over 400 million objects written to S3 per day. Total storage is in the exabyte range.

To manage costs, Instagram uses S3’s storage tiers. Recent photos (last 30 days) stay in S3 Standard for fast access. Older photos migrate to S3 Infrequent Access, and very old photos to Glacier — still accessible but with higher retrieval latency.

Instagram Stories

Stories — ephemeral content that disappears after 24 hours — launched in 2016 and quickly became Instagram’s most-used feature, with over 500 million daily users.

Why Stories Are Architecturally Different

Stories differ from feed posts in critical ways:

• Ephemeral — auto-deleted after 24 hours, so the storage model is fundamentally different
• Sequential — viewed in order (slide-through), not as a ranked feed
• High write volume — users post multiple stories per day, far more than feed posts
• Different access pattern — stories from the last 24 hours are hot; everything else is irrelevant

Stories Architecture:

Storage Layer:
  -> Stories stored in Cassandra (write-optimized)
  -> TTL set to 24 hours (auto-deletion, no garbage collection)
  -> Media files in S3 with lifecycle policy

Stories Tray (the row of circles at top):
  -> For each user, compute which follows have active stories
  -> Sort by: unseen first, then recency, then engagement affinity
  -> Cache in Redis with 5-minute TTL

Viewing Flow:
  1. User taps a story circle
  2. Fetch story list for that user from Cassandra
  3. Prefetch next 2-3 users' stories (anticipating swipe)
  4. Mark as viewed (write to Cassandra)
  5. Update view count (async, eventual consistency is fine)

Cassandra’s built-in TTL feature is perfect for Stories. Instead of running batch jobs to delete expired content, each story row is written with a 24-hour TTL. Cassandra’s compaction process automatically removes expired data. This eliminates an entire class of cleanup infrastructure.

Stories Viewer List

When you check who viewed your story, Instagram must maintain a per-story viewer list that can grow to millions for popular accounts. This is stored as an append-only log in Cassandra, partitioned by story ID. The viewer list UI paginates through this log, showing the most recent viewers first.

Scaling Django to 2B+ Users

Instagram is the largest known Django deployment in the world. Their engineering team has spoken extensively about how they scaled a framework designed for small-to-medium web applications to serve billions of users.

How They Did It

The approach was pragmatic: keep Django for what it does well (request handling, ORM, URL routing), and bypass it for what it does not.

Django Scaling Strategies:

1. Disable what you don't need:
   -> Turned off Django middleware not in use
   -> Removed unused Django apps from INSTALLED_APPS
   -> Custom stripped-down settings for API servers

2. Async where it matters:
   -> Background tasks offloaded to Celery workers
   -> Image processing never blocks the request path
   -> Feed writes are asynchronous

3. Aggressive caching:
   -> Memcached (via Django's cache framework) for
      rendered responses, query results, session data
   -> Redis for feed caches (sorted sets of post IDs)
   -> CDN for all static assets and media

4. Connection pooling:
   -> PgBouncer sits between Django and PostgreSQL
   -> Django's per-request connection model would
      overwhelm PostgreSQL without pooling
   -> Each PgBouncer instance handles 10,000+ connections
      funneled into ~100 actual PostgreSQL connections

5. Multiple Django process models:
   -> uWSGI with multiple workers per server
   -> Separate server pools for web, API, and background tasks

Cython Optimization

Instagram’s team selectively compiled performance-critical Django code with Cython, converting Python to C extensions. This yielded 10-30 percent speedups on hot paths without rewriting the application logic. The approach targets the 5 percent of code that accounts for 50 percent of CPU time.

Database Architecture: PostgreSQL Sharding

Instagram stuck with PostgreSQL long after conventional wisdom would have suggested switching to NoSQL. They made it work through aggressive sharding and connection pooling.

Sharding Strategy

Instagram shards PostgreSQL by user ID. Each shard is a separate PostgreSQL instance containing a slice of the user space. The application layer determines which shard to query based on the user ID.

# Simplified shard routing
SHARD_COUNT = 4096

def get_shard(user_id):
    """Route user to their PostgreSQL shard."""
    shard_id = user_id % SHARD_COUNT
    return shard_connections[shard_id]

def get_user_posts(user_id):
    shard = get_shard(user_id)
    return shard.query(
        "SELECT * FROM posts WHERE user_id = %s ORDER BY created_at DESC",
        [user_id]
    )

ID Generation

Instagram needed globally unique IDs that could be generated independently on any shard without coordination. They built a system similar to Twitter’s Snowflake, using PostgreSQL’s PL/pgSQL:

-- Instagram's ID generation (actual schema they published)
-- 41 bits: timestamp (milliseconds since custom epoch)
-- 13 bits: shard ID
-- 10 bits: auto-incrementing sequence

CREATE OR REPLACE FUNCTION insta_next_id(
    OUT result bigint
) AS $$
DECLARE
    our_epoch bigint := 1314220021721;
    seq_id bigint;
    now_ms bigint;
    shard_id int := 5;  -- unique per shard
BEGIN
    SELECT nextval('insta_id_seq') % 1024 INTO seq_id;
    SELECT FLOOR(EXTRACT(EPOCH FROM clock_timestamp()) * 1000)
        INTO now_ms;
    result := (now_ms - our_epoch) << 23;
    result := result | (shard_id << 10);
    result := result | (seq_id);
END;
$$ LANGUAGE plpgsql;

This generates roughly-time-ordered IDs (the timestamp is in the most significant bits), which keeps B-tree indexes efficient. IDs are globally unique because the shard ID is embedded.

PgBouncer: Connection Pooling

Django opens a new database connection per request by default. With thousands of requests per second, this would overwhelm PostgreSQL (which creates a process per connection). PgBouncer sits between Django and PostgreSQL, maintaining a pool of actual connections.

Without PgBouncer:
  1000 Django workers -> 1000 PostgreSQL connections
  (PostgreSQL struggles above ~500 connections)

With PgBouncer:
  1000 Django workers -> PgBouncer -> 100 PostgreSQL connections
  (PgBouncer multiplexes, reuses connections)

Instagram runs PgBouncer in transaction mode, where a PostgreSQL connection is assigned to a Django worker only for the duration of a single transaction, then returned to the pool.

Explore Page: ML-Powered Discovery

The Explore page is Instagram’s content discovery engine, serving personalized content from accounts you do not follow. It is responsible for a significant portion of new follower growth on the platform.

Architecture

Explore Page Pipeline:

1. Candidate Generation
   -> Accounts similar to ones you follow
   -> Posts liked by people with similar taste
   -> Trending content in your region
   -> ~10,000 candidate posts

2. First-Pass Ranking (lightweight model)
   -> Score candidates with a fast model
   -> Features: post engagement rate, author quality score,
      content type, freshness
   -> Narrow to ~500 posts

3. Second-Pass Ranking (heavy model)
   -> Deep neural network
   -> Features: your full interaction history,
      visual features of the image (CNN embeddings),
      caption NLP features
   -> Score: P(engage | user, post)

4. Diversity and Policy Filters
   -> Ensure topic diversity (not all food photos)
   -> Remove near-duplicate content
   -> Apply content policy (no misinformation, violence)
   -> Final ~50 posts for initial page load

5. Real-Time Re-Ranking
   -> As user scrolls and engages, re-rank remaining candidates
   -> Adapt to session-level signals

Technology Stack Summary

Language:      Python (Django), some C++ and Go for infra
Framework:     Django (heavily customized)
Databases:     PostgreSQL (sharded, primary store),
               Cassandra (Stories, activity feeds),
               Redis (feed cache, sessions),
               Memcached (general caching)
Task Queue:    Celery + RabbitMQ
Storage:       Amazon S3 (media files)
CDN:           Facebook/Meta CDN
ML:            PyTorch (ranking models), Caffe2
Monitoring:    Custom dashboards, Sentry for errors
Deployment:    Custom CI/CD on Meta infrastructure

Key Takeaways

Instagram’s architecture proves that technology choice is less important than engineering discipline:

• Boring technology scales. Django and PostgreSQL were not designed for 2 billion users. But with sharding, connection pooling, caching, and async processing, they got there. Choosing well-understood tools meant fewer surprises at 3 AM.
• The feed problem defines social media architecture. Fanout on write vs read, celebrity handling, and ranking — these decisions cascade through every other system. Get the feed strategy right, and everything else follows.
• Ephemeral content needs different storage. Stories’ 24-hour TTL makes Cassandra a better fit than PostgreSQL. Do not force one database to serve all access patterns.
• Async everything that is not user-facing. Image processing, feed fanout, push notifications — none of these should block the upload response. The user sees “posted” in under a second; the system does the rest in the background.
• Connection pooling is not optional at scale. PgBouncer turns PostgreSQL from a system that breaks at 500 connections into one that handles tens of thousands of concurrent Django workers.

When you scroll through Instagram and your feed loads in under a second, that speed comes from a pre-computed feed cache in Redis, images served from a nearby CDN edge, and a Django application that has been carefully tuned to do less work per request — proving that you do not need to rewrite everything in Go or Rust to build one of the largest applications in the world.