System Design: Designing a Distributed BLOB Store
An object store (BLOB store) is a fundamental building block of cloud infrastructure. Unlike a file system, it provides a simple interface (PUT, GET, DELETE) to store large, unstructured data (images, videos, backups). Designing this at scale is an exercise in data durability and efficient storage.
1. Core Requirements
- High Durability: Data must survive node and disk failures (aiming for 11 nines).
- Infinite Scalability: Support exabytes of data across thousands of nodes.
- Low Latency: Fast retrieval of objects regardless of their size.
- Consistency: Strong consistency for object metadata.
2. The Data Plane: Erasure Coding
Storing three full replicas of every object is too expensive.
- Erasure Coding (Reed-Solomon): Instead of replication, we break an object into data blocks and calculate parity blocks.
- Durability: If drives fail, the original object can still be reconstructed using the remaining blocks.
- Efficiency: Much higher storage efficiency compared to 3x replication.
3. The Metadata Plane: Hierarchical Indexing
While raw data is in BLOBs, the metadata (file size, location, owner) needs to be queried.
- Metadata Store: A distributed NoSQL key-value store (like DynamoDB or a custom sharded Cassandra cluster).
- Index: Stored as . The value contains a list of physical block locations on the storage nodes.
4. Addressing Hotspots: Consistent Hashing
To prevent any single storage node from becoming a bottleneck, use Consistent Hashing (discussed in our earlier article) to distribute objects across storage nodes uniformly.
5. Summary
A BLOB store is designed to turn hardware failure into a non-event. By using Erasure Coding for efficient durability and a Distributed Metadata Store for fast object lookup, you can build a storage system that scales horizontally with zero impact on reliability.
