System Design: Design an Object Storage System

Object storage is the backbone of modern cloud infrastructure. Amazon S3 alone stores over 350 trillion objects. Unlike file systems (hierarchical) or block storage (fixed-size blocks), object storage provides a flat namespace of immutable objects accessed via a simple API: PUT, GET, DELETE. This simplicity is what makes it scalable to exabyte levels.

Designing an object storage system reveals fundamental lessons about metadata management, data placement, durability engineering, and multi-tenant isolation.

Functional Requirements

• Store and retrieve objects (binary blobs) up to 5 TB in size.
• Organize objects into buckets (logical containers) with globally unique names.
• Support listing objects within a bucket with prefix filtering and pagination.
• Provide access control at the bucket and object level.
• Support metadata (content type, custom headers) attached to each object.
• Support multipart uploads for large objects.

Non-Functional Requirements

• Durability: 11 nines (99.999999999%) — losing an object should be astronomically unlikely.
• Availability: 99.99% for reads, 99.9% for writes.
• Scalability: store petabytes of data across thousands of storage nodes.
• Throughput: handle millions of requests per second globally.

High-Level Architecture

The system separates into three layers:

1. API layer — stateless web servers that handle HTTP requests, authenticate callers, and route operations.
2. Metadata layer — a distributed database storing bucket and object metadata (name, size, location, ACLs).
3. Data layer — storage nodes holding the actual object bytes, organized into chunks.

Client -> [API Gateway] -> [Metadata Service] -> [Metadata DB]
                        -> [Data Service] -> [Storage Nodes]

PUT flow: the API server writes object data to the data layer, receives the storage locations, then writes the metadata record. The object is not visible to readers until the metadata record is committed.

GET flow: the API server looks up the metadata to find the object’s storage locations, then reads the data directly from the storage nodes.

Metadata Service

The metadata service maps (bucket, key) to the object’s storage locations and properties. This is the brain of the system.

Data model:

buckets
  name          VARCHAR PRIMARY KEY (globally unique)
  owner_id      UUID
  created_at    TIMESTAMP
  acl           JSONB

objects
  bucket        VARCHAR
  key           VARCHAR
  version_id    UUID (for versioned buckets)
  size          BIGINT
  content_type  VARCHAR
  checksum      VARCHAR
  chunk_locations  JSONB  (list of chunk IDs and storage node addresses)
  created_at    TIMESTAMP
  PRIMARY KEY (bucket, key, version_id)

Scaling the metadata store. The metadata database must handle billions of objects. Shard by bucket name (hash partitioning) to distribute load across database nodes. Within a bucket, objects are stored sorted by key to support prefix listing efficiently.

Use a distributed database like CockroachDB, TiDB, or a custom sharded PostgreSQL setup. The metadata layer is the most critical component — if it is unavailable, no reads or writes can proceed.

Data Storage Layer

The data layer stores raw bytes. Large objects are split into fixed-size chunks (for example, 64 MB). Each chunk is stored redundantly across multiple storage nodes.

Storage node design. Each storage node manages a set of local disks. It exposes a simple API: write_chunk(chunk_id, data), read_chunk(chunk_id), delete_chunk(chunk_id). Internally, it stores chunks as files on a local filesystem, organized by chunk ID prefix for directory fan-out.

Each storage node reports its capacity, disk health, and chunk inventory to a central placement service via heartbeats.

Data Placement

When a new chunk needs to be stored, the placement service decides which storage nodes should hold it. The algorithm must balance multiple constraints:

• Spread replicas across failure domains: place each replica on a different rack, or ideally a different availability zone, so that a single hardware failure does not lose all copies.
• Balance disk utilization: avoid hot spots by distributing chunks proportionally to available capacity.
• Minimize data movement: when nodes join or leave the cluster, move as little data as possible.

Consistent hashing with virtual nodes works well here. Map each storage node to multiple points on a hash ring (proportional to its capacity). Hash the chunk ID to find its position on the ring. Walk clockwise to find N distinct nodes in different failure domains. This naturally balances load and minimizes redistribution when the cluster changes.

Durability: Replication vs. Erasure Coding

To achieve 11 nines of durability, data must survive disk failures, node failures, and even datacenter losses.

Triple replication. Store three complete copies of each chunk on different nodes. Simple and fast for reads (any copy serves the request) but expensive: 3x storage overhead.

Erasure coding. Split each chunk into k data fragments and compute m parity fragments. Any k of the k+m fragments can reconstruct the original data. For example, a (10, 4) scheme splits a 64 MB chunk into 10 data fragments (6.4 MB each) and 4 parity fragments, for a total of 14 fragments. Store each on a different node. You can lose any 4 nodes and still reconstruct the data, with only 1.4x storage overhead instead of 3x.

Trade-off. Erasure coding saves storage but increases read latency (you must read from k nodes and decode) and repair complexity (reconstructing a lost fragment requires reading k other fragments). Use erasure coding for cold data and replication for hot data.

Write Path

1. Client sends PUT /bucket/key with the object body.
2. The API server authenticates the request and checks bucket-level permissions.
3. For large objects, the client uses multipart upload: split into parts, upload each part independently, then complete the upload with a manifest.
4. The API server splits the object into chunks.
5. For each chunk, the placement service selects target storage nodes.
6. The API server streams the chunk data to all target nodes in parallel. It waits for a write quorum (for example, 2 of 3 replicas) before proceeding.
7. After all chunks are written, the API server atomically commits the metadata record (bucket, key, chunk locations, checksum).
8. The object becomes visible to readers.

Write-ahead logging. If the API server crashes between writing chunks and committing metadata, the chunks become orphaned. A background garbage collector periodically scans for chunks not referenced by any metadata record and deletes them.

Read Path

1. Client sends GET /bucket/key.
2. The API server queries the metadata service for the object’s chunk locations.
3. For each chunk, the API server reads from the nearest or least-loaded replica.
4. The API server streams the reconstructed object to the client.
5. For range requests (Range: bytes=1000-1999), the API server calculates which chunk contains the requested byte range and reads only that chunk.

Consistency Model

Object storage typically provides read-after-write consistency: after a successful PUT, any subsequent GET returns the new object. This is achieved by committing the metadata record synchronously before returning success to the client.

For listing consistency, ensure the metadata store provides consistent reads. If the metadata store uses asynchronous replication, listing may not immediately reflect a recently written object. Use synchronous replication or read from the primary for listing operations.

Concurrent writes to the same key. Last-writer-wins based on timestamp. Alternatively, support versioning: each write creates a new version, and all versions are retained until explicitly deleted.

Multi-Tenant Isolation

Object storage serves many customers on shared infrastructure.

Namespace isolation. Bucket names are globally unique. Each bucket belongs to one account. All authorization checks verify that the caller owns or has been granted access to the bucket.

Quota enforcement. Track storage usage per account. Enforce quotas at the API layer before accepting writes. Maintain a near-real-time usage counter using asynchronous aggregation.

Performance isolation. A single tenant should not monopolize storage node bandwidth. Implement per-tenant rate limiting at the API layer and per-node I/O throttling to ensure fair sharing.

Data isolation. Encrypt each object with a unique data encryption key (DEK), which is itself encrypted with the tenant’s key encryption key (KEK). This provides cryptographic isolation: even if a bug exposes raw data, cross-tenant access is prevented.

Garbage Collection and Compaction

Deleted objects leave behind unreferenced chunks. A background garbage collector runs periodically:

1. Scan the metadata store for all referenced chunk IDs.
2. Scan storage nodes for all stored chunk IDs.
3. Delete chunks that exist on storage nodes but are not referenced by any metadata record.

This is expensive at scale. Optimize by keeping a bloom filter of referenced chunks in memory and checking it before deleting.

Monitoring

• Durability metrics: number of chunks below the target replica count. This should be zero; any nonzero value triggers immediate repair.
• Storage utilization: per-node and per-cluster disk usage.
• Request latency: p50, p95, p99 for GET and PUT operations.
• Error rate: percentage of 5xx responses.
• Repair rate: number of chunks being repaired per second. During node replacement, this spikes and then returns to baseline.

Summary

An object storage system separates metadata (small, indexed, strongly consistent) from data (large, replicated or erasure-coded, eventually consistent). The critical design decisions are how you place data across failure domains (consistent hashing with failure-domain awareness), how you achieve durability (replication for hot data, erasure coding for cold), and how you manage the metadata layer (the single most important component to get right). Start with triple replication and a sharded metadata store, then add erasure coding as storage costs demand optimization.