System Design: Designing an Object Store (Amazon S3)

An Object Store is a distributed storage system designed to store massive amounts of unstructured data (Photos, Videos, Backups). Unlike a file system, it provides a simple HTTP interface and treats everything as an "Object."

1. Core Requirements

• Scalability: Storing exabytes of data across millions of hard drives.
• Durability (11 9s): The probability of losing a file must be effectively zero.
• Availability: Always accessible via HTTP.
• Multi-tenancy: Securely isolating data between different users/buckets.

2. Objects vs. Files

• Flat Namespace: No folders or directories (though they can be simulated with prefixes like images/).
• Immutable: You don't "update" an object; you overwrite it with a new version.
• Metadata: Custom key-value pairs stored alongside the data.

3. Durability Secret: Erasure Coding

Storing 3 full copies of every file (Replication) is too expensive at exabyte scale.

• The Solution: Erasure Coding (Reed-Solomon).
• How it works: A file is split into K data blocks, and M parity (calculation) blocks are added.
• The Result: Even if you lose any M blocks out of the total (K+M), you can mathematically reconstruct the original file.
• Cost: Much lower overhead than 3x replication while providing even higher durability.

4. Metadata Architecture

Searching for billions of objects by their name (Key) requires a high-performance index.

• Store: Use a distributed NoSQL store like DynamoDB or a customized LSM-tree based key-value store.
• The Key: BucketName + ObjectName.
• The Value: A pointer to the physical location of the data blocks on disk.

5. Storage Nodes and Placement

• Physical Layout: Data centers are divided into Availability Zones (AZs).
• Placement Policy: To survive a total region failure, data blocks for a single object are spread across multiple disks, multiple racks, and multiple AZs.

6. Ensuring Data Integrity: Background Scrubbing

Hard drives eventually fail or develop "bit rot."

• The Process: A background worker continuously reads random blocks of data, calculates their checksum, and compares it to the stored checksum.
• Repair: If a block is corrupt, the system uses Erasure Coding to reconstruct it from the remaining healthy blocks.

Summary

The engineering of an object store is a battle against Hardware Entropy. By leveraging Erasure Coding for efficiency and proactive background scrubbing for durability, you can build a system that stores the world's data with near-absolute reliability.