MongoDB Internals: Under the Hood
MongoDB has evolved from a simple JSON-like store into a sophisticated distributed database. Most of its power comes from its default storage engine, WiredTiger, and its robust replication model.
1. WiredTiger Storage Engine
Since MongoDB 3.2, WiredTiger has been the default engine. It provides high performance through several key features:
- Document-Level Concurrency: Unlike the old MMAPv1 engine which had collection-level locking, WiredTiger uses optimistic concurrency control with document-level locking.
- Checkpoints: WiredTiger writes data to disk every 60 seconds or after 2GB of data changes. This ensures durability while minimizing disk I/O.
- Compression: It supports Snappy and Zlib compression for both data and indexes, significantly reducing storage footprints.
- Journaling: To ensure data isn't lost between checkpoints, WiredTiger uses a write-ahead log (journal).
2. Replication and the Oplog
MongoDB high availability is managed through Replica Sets.
- Primary: Receives all writes.
- Secondaries: Replicate the Primary's state.
- Oplog (Operations Log): A capped collection that stores a rolling record of all data-modifying operations. Secondaries fetch this log to stay in sync.
Election Process
When a Primary fails, an election occurs. MongoDB uses a protocol based on Raft to ensure that only a secondary with the most recent data can be elected, preventing data rollback.
3. Sharding: Horizontal Scaling
Sharding allows MongoDB to scale beyond the limits of a single server by distributing data across multiple clusters.
- Shard Key: The most critical decision. It determines how data is partitioned.
- Chunk Splitting: As a shard grows, MongoDB splits "chunks" of data to maintain balance.
- Balancer: A background process that migrates chunks from over-utilized shards to under-utilized ones.
4. Query Optimization
MongoDB uses a Rule-based and Cost-based Optimizer.
- It analyzes several candidate indexes.
- It runs them in parallel for a short time to see which one performs best.
- It caches the "winning" plan for subsequent identical queries.
Summary
Understanding WiredTiger and the Oplog is essential for any developer looking to scale MongoDB. By leveraging document-level locking and intelligent sharding, MongoDB can handle massive workloads while maintaining developer productivity.
