System Design: Designing Multi-Region Active-Active Shopping Carts

When building an e-commerce platform operating at a global scale, the shopping cart is one of the most business-critical systems. A single minute of shopping cart downtime directly translates to millions of dollars in lost revenue.

To survive complete regional cloud outages and provide low-latency operations to users worldwide, the shopping cart must be designed using a Multi-Region Active-Active Architecture.

This case study analyzes the architecture of a highly available, active-active shopping cart engine, detailing geographical traffic routing, Cassandra wide-column data modeling, and mathematical conflict resolution using Conflict-Free Replicated Data Types (CRDTs).

1. Requirements & Core Constraints

Designing a global active-active shopping cart system requires balancing high write availability with eventual consistency trade-offs.

Functional Constraints

• Multi-Region Shopping Cart Management: Users must be able to view, add, update, and remove items from their carts dynamically.
• Unified Multi-Device Syncing: A user adding an item on their phone must see it immediately on their laptop once both devices sync.
• Seamless Checkout Hand-off: The aggregated cart state must be cleanly frozen and dispatched to the order processing system at checkout.
• Cross-Region Cart Convergence: Cart updates made concurrently in different geographical regions must merge deterministically without dropping user items.

Non-Functional SLAs

• Ultra-High Availability: Target $99.999%$ uptime SLA for cart operations, surviving total regional connectivity loss.
• Strict Latency Limits: Cart read and write latency must be $\le 100\text{ms}$ at the 99th percentile (p99) globally.
• AP-Mode Tolerant: Under regional network partitions, the system must prioritize write availability over strong consistency (following the CAP theorem).
• Eventually Consistent: Cart merges must guarantee that all replicas converge to the exact same state once synchronization lag settles.

Back-of-the-Envelope Estimation

1. Active Users and QPS Sizing

• Monthly Active Users (MAU): $500,000,000$ (500 Million) global users.
• Daily Active Users (DAU): $100,000,000$ (100 Million) active users.
• Average Cart Operations per User/Day: Assume an active user performs 5 cart additions/updates per day.
• Daily Cart Write Volume: $$\text{Daily Writes} = 100,000,000 \times 5 = 500,000,000 \text{ writes/day}$$
• Average Write QPS: $$\text{Average QPS} = \frac{500,000,000 \text{ writes}}{86,400 \text{ seconds}} \approx 5,787 \text{ writes/sec}$$
• Peak Write QPS: Sized for a peak factor of $5\times$ to accommodate global flash sale spikes: $$\text{Peak Write QPS} = 5,787 \times 5 \approx 28,935 \text{ writes/sec}$$

2. Network Replication Bandwidth

• Cart Metadata Payload: Average serialized cart payload size is $2\text{ KB}$ containing item listings, user details, and synchronization vector clocks.
• Regional Write Capacity: If traffic is distributed across 3 global hubs (US-West, EU-Central, AP-Southeast), each region processes roughly one-third of the peak load: $$\text{Regional Peak Write QPS} \approx 10,000 \text{ writes/sec}$$
• WAN Replication Ingress/Egress: Each local region must asynchronously replicate its local writes to the other two regions over the Wide Area Network (WAN): $$\text{Replication Egress Traffic} = 10,000 \text{ writes/sec} \times 2 \text{ KB} = 20 \text{ MB/s} \approx 160 \text{ Mbps}$$

2. API Design & Core Contracts

The shopping cart API exposes low-latency stateless write endpoints coupled with version vectors for conflict detection.

1. Add Item to Cart

POST /api/v1/cart/items Triggered when a user clicks "Add to Cart" on a product detail page.

Request Headers:

Content-Type: application/json
X-User-Fingerprint: fp_992e88a1b
X-Client-Timestamp: 1782236500000

Request Payload:

{
  "user_id": "usr_99f8e7d6c5",
  "item_id": "itm_shirt123",
  "quantity": 2,
  "unit_price": 29.99,
  "version_vector": {
    "us-west": 45,
    "eu-central": 12
  }
}

Response Payload (200 OK):

{
  "cart_id": "crt_88e7d6c5",
  "user_id": "usr_99f8e7d6c5",
  "items": [
    {
      "item_id": "itm_shirt123",
      "quantity": 2,
      "added_at": 1782236500000
    }
  ],
  "version_vector": {
    "us-west": 46,
    "eu-central": 12
  }
}

2. Remove Item from Cart

DELETE /api/v1/cart/items Executes when a user deletes an item from their active checkout list.

Request Payload:

{
  "user_id": "usr_99f8e7d6c5",
  "item_id": "itm_shirt123",
  "version_vector": {
    "us-west": 46,
    "eu-central": 12
  }
}

Response Payload (200 OK):

{
  "cart_id": "crt_88e7d6c5",
  "user_id": "usr_99f8e7d6c5",
  "items": [],
  "version_vector": {
    "us-west": 47,
    "eu-central": 12
  }
}

3. High-Level Design (HLD)

To achieve high availability and low latency, we deploy the system in a multi-region active-active configuration. All regions accept read and write traffic, asynchronously replicating state changes across the WAN.

graph TD
    %% User Devices
    UserPhone[User's Mobile Phone] -->|1. DNS Anycast Route| Anycast[DNS Anycast Router]
    UserLaptop[User's Laptop] -->|1. DNS Anycast Route| Anycast
    
    %% Traffic Management
    Anycast -->|Geo Routing| LB_US[US-West Load Balancer]
    Anycast -->|Geo Routing| LB_EU[EU-Central Load Balancer]

    subgraph Region_US["US-West Region (Active)"]
        LB_US -->|Stateless Routes| Gateway_US[US API Gateway Fleet]
        Gateway_US -->|Read/Write Carts| App_US[US Cart Service App]
        App_US <-->|Session Store| Redis_US[("Local Redis Cluster")]
        App_US <-->|Persist State| Cassandra_US[("Local Cassandra Ring (DC1)")]
    end

    subgraph Region_EU["EU-Central Region (Active)"]
        LB_EU -->|Stateless Routes| Gateway_EU[EU API Gateway Fleet]
        Gateway_EU -->|Read/Write Carts| App_EU[EU Cart Service App]
        App_EU <-->|Session Store| Redis_EU[("Local Redis Cluster")]
        App_EU <-->|Persist State| Cassandra_EU[("Local Cassandra Ring (DC2)")]
    end

    %% Cross-Region Replication Pipelines
    Cassandra_US <-->|Asynchronous Multi-DC Replication (WAN)| Cassandra_EU
    Redis_US <-->|Asynchronous Cache Sync| Redis_EU

    classDef database fill:#0d3b66,stroke:#f4d35e,stroke-width:2px,color:#fff;
    classDef cluster fill:#2e0f38,stroke:#f4d35e,stroke-width:2px,color:#fff;
    classDef client fill:#3d5a80,stroke:#293241,stroke-width:2px,color:#fff;
    classDef loadbalancer fill:#ee6c4d,stroke:#293241,stroke-width:2px,color:#fff;
    
    class Cassandra_US,Cassandra_EU,Redis_US,Redis_EU database;
    class Gateway_US,Gateway_EU,App_US,App_EU cluster;
    class UserPhone,UserLaptop client;
    class Anycast,LB_US,LB_EU loadbalancer;

End-to-End Architectural Workflows

1. Low-Latency Geo-Routing and Edge Ingestion

1. Dynamic Routing: When a user updates their cart, the request hits a DNS Anycast routing proxy. The user is mapped to the physically nearest active region (e.g., US-West) via BGP routes.
2. TLS Termination: The edge API Gateway terminates the secure connection in under $30\text{ms}$ and proxies the request to the regional Cart Microservices cluster.
3. Session Cache Check: The service checks the Local Redis Cluster to retrieve the cached cart state. If it is a cache hit, the operation completes in $< 2\text{ms}$.

2. Local Database Write and WAN Asynchronous Replication

1. Local Persistent Commit: The local application writes the updated cart payload to the Local Cassandra Ring using a LOCAL_QUORUM consistency level. This requires acknowledgment from a majority of nodes within the local data center (e.g., 2 out of 3), providing durability in $< 10\text{ms}$ without waiting for cross-region network links.
2. Asynchronous Cross-DC Replication: Cassandra's native gossip-based multi-datacenter replication runs in the background. State changes are serialized and transmitted over secure WAN pipes to the companion datacenter (EU-Central) to synchronize updates asynchronously.
3. Cache Invalidation: Upon receiving a replicated write, the remote Cassandra node triggers a database event listener that invalidates the corresponding cart entry in the remote Redis cluster, preparing it for the next user request in that region.

4. Low-Level Design (LLD) & Data Models

Database Selection Rationale

An active-active shopping cart requires a database engine designed to survive partition events while continuing to accept writes.

CQL Schema Design (Apache Cassandra)

To model the shopping cart, we leverage wide-column partitions keyed by user_id. Each individual item within a user's cart is mapped to a distinct row, utilizing Cassandra clustering keys for sorting.

CREATE KEYSPACE IF NOT EXISTS csp_shopping_cart
WITH replication = {
    'class': 'NetworkTopologyStrategy',
    'us-west': 3,
    'eu-central': 3
};

-- Persistent Multi-Region Shopping Cart Table
CREATE TABLE csp_shopping_cart.user_carts (
    user_id text,
    item_id text,
    quantity int,
    unit_price decimal,
    added_at timestamp,
    tombstone_ids set<text>, -- Set of unique IDs tracking removed items to resolve deletions
    version_vector map<text, bigint>, -- Regional lamport clock mappings for merge conflicts
    PRIMARY KEY (user_id, item_id)
) WITH CLUSTERING ORDER BY (item_id ASC);

5. Stateful Conflict Resolution: CRDTs vs. LWW

In an active-active setup, a user might add an item to their cart on their phone (connected to us-west) and immediately remove it on their laptop (connected to eu-central via a VPN). Since writes can occur concurrently in different regions, we need a deterministic strategy to merge the states.

1. The Pitfalls of Last-Write-Wins (LWW)

In a Last-Write-Wins (LWW) model, the system uses the write's physical wall-clock timestamp to resolve conflicts. While simple, this approach has two fatal flaws:

1. Clock Skew: Even with Network Time Protocol (NTP) synchronization, physical server clocks drift by milliseconds. A server with a slightly fast clock can overwrite a newer write from a server with a slow clock.
2. Dropped Updates: If a user performs two rapid operations in different regions, LWW will drop the older update entirely, potentially deleting items the user intended to purchase.

2. Conflict-Free Replicated Data Types (CRDTs): Observed-Remove Set (OR-Set)

To resolve conflicts reliably, we use an Observed-Remove Set (OR-Set) CRDT. An OR-Set assigns a unique cryptographic tag to every item added to the cart.

• Add Operation: When an item is added, a unique tracking ID is generated and added to the element's set of active tags.
• Remove Operation: When an item is removed, all active tracking IDs observed for that element are added to a persistent Tombstone Set.
• Merge Operation: When merging two replicas, the final cart is computed as: $$\text{Active Items} = \text{All Added Elements} \setminus \text{Items in the Tombstone Set}$$

graph TD
    subgraph US_Region["US-West Node"]
        AddA1["Add Item A (Tag: T1)"]
    end

    subgraph EU_Region["EU-Central Node"]
        AddA2["Add Item A (Tag: T2)"]
        RemoveA1["Remove Item A (Observes T2)"]
    end

    US_Region -->|Replicate Add A (T1)| MergeNode["Replication Merge Protocol"]
    EU_Region -->|Replicate Add A (T2) & Remove (T2)| MergeNode

    MergeNode -->|Calculates: {T1, T2} - {T2} = {T1}| FinalCart["Final Cart Contains Item A (Tag: T1)"]

Compilable Java OR-Set Shopping Cart Merger

Below is a complete, compilable implementation of a state-based Observed-Remove Set (OR-Set) designed to merge concurrent shopping cart edits:

package com.codesprintpro.shoppingcart.crdt;

import java.io.Serializable;
import java.util.*;

public class ShoppingCartMerger {

    // Unique Identifier for Cart Items with Tagged Additions
    public static class CartElement implements Serializable {
        public final String itemId;
        public final String additionId; // UUID generated at local "add" execution

        public CartElement(String itemId, String additionId) {
            this.itemId = itemId;
            this.additionId = additionId;
        }

        @Override
        public boolean equals(Object o) {
            if (this == o) return true;
            if (!(o instanceof CartElement)) return false;
            CartElement that = (CartElement) o;
            return Objects.equals(itemId, that.itemId) && Objects.equals(additionId, that.additionId);
        }

        @Override
        public int hashCode() {
            return Objects.hash(itemId, additionId);
        }
    }

    // Stateful CRDT Representation of the Shopping Cart
    public static class ORSetShoppingCart implements Serializable {
        public final Set<CartElement> additions = new HashSet<>();
        public final Set<String> tombstones = new HashSet<>(); // Tracks deleted additionIds
        public final Map<String, Integer> itemQuantities = new HashMap<>();

        // Add item to local cart
        public void addItem(String itemId, int quantity) {
            String uniqueAddId = UUID.randomUUID().toString();
            additions.add(new CartElement(itemId, uniqueAddId));
            itemQuantities.put(itemId, itemQuantities.getOrDefault(itemId, 0) + quantity);
        }

        // Remove item from local cart (tombstones all observed additionIds)
        public void removeItem(String itemId) {
            for (CartElement elem : additions) {
                if (elem.itemId.equals(itemId)) {
                    tombstones.add(elem.additionId);
                }
            }
            itemQuantities.remove(itemId);
        }

        // Merge two concurrent shopping carts deterministically
        public ORSetShoppingCart merge(ORSetShoppingCart remoteCart) {
            ORSetShoppingCart merged = new ORSetShoppingCart();
            
            // 1. Merge Tombstones (Union of all deletions)
            merged.tombstones.addAll(this.tombstones);
            merged.tombstones.addAll(remoteCart.tombstones);

            // 2. Merge Additions (Union of all additions)
            Set<CartElement> allAdditions = new HashSet<>(this.additions);
            allAdditions.addAll(remoteCart.additions);

            // 3. Filter Additions against Tombstones
            for (CartElement addition : allAdditions) {
                if (!merged.tombstones.contains(addition.additionId)) {
                    merged.additions.add(addition);
                }
            }

            // 4. Resolve Quantities (Maximum of concurrent quantities for simplicity)
            Set<String> activeItemIds = new HashSet<>();
            for (CartElement active : merged.additions) {
                activeItemIds.add(active.itemId);
            }

            for (String itemId : activeItemIds) {
                int localQty = this.itemQuantities.getOrDefault(itemId, 0);
                int remoteQty = remoteCart.itemQuantities.getOrDefault(itemId, 0);
                merged.itemQuantities.put(itemId, Math.max(localQty, remoteQty));
            }

            return merged;
        }
    }
}

6. Scaling Challenges & System Bottlenecks

Operating an active-active database ring across WAN links introduces complex failure patterns. Here is how we address these bottlenecks:

1. Cassandra Write-Path Replication Lag & Checkouts

• The Bottleneck: While cart additions are eventually consistent, executing a checkout requires a consistent view of the cart. If a user clicks "Checkout" in a region that hasn't received the latest asynchronous replicates from another region, they might check out with stale items.
• The Mitigation: Quorum Checkout Elevation.
  • Standard cart updates are written using LOCAL_QUORUM (fast, local consensus).
  • When the user transitions to the checkout phase, the API Gateway elevates the read consistency level to QUORUM across data centers (requiring agreement from a majority of global nodes, e.g., 4 out of 6 nodes across the US and EU).
  • This cross-DC read forces Cassandra to resolve replication lag on the fly, guaranteeing the checkout service processes the most up-to-date cart state.

2. The "New-Enemy" Physical Write Race Condition

• The Bottleneck: In a true multi-device scenario, a user might trigger rapid additions and deletions from two separate devices hitting different regions simultaneously. If a write is acknowledged in US-West while a concurrent delete is acknowledged in EU-Central, network delays can cause them to arrive out of order at their destination replicas.
• The Mitigation: Version Map Clock Anchoring. We embed a regional monotonic sequence vector (version map) in the Cassandra row payload. The merge engine compares vector histories. If a replica receives a change that lacks the preceding causal steps, the change is held in a buffer until the dependent state arrives, preventing out-of-order data corruption.

7. Technical Trade-offs & Consistency Models

1. CAP Theorem AP vs. CP Architectural Alignment

2. tombstone Garbage Collection Overhead

8. Resilience & Failure Scenarios

1. Complete Regional Cloud Outage (Region Failover)

If an entire cloud region (e.g., US-West) goes offline, all services and database nodes in that datacenter become unavailable.

• Recovery Protocol:
  • The Anycast router detects the regional health check failure and updates its routing tables within 10 seconds, redirecting all incoming traffic to EU-Central.
  • EU-Central accepts all read and write requests immediately.
  • Since the database uses masterless multi-region replication, the EU-Central data center already contains the latest replicated cart states (subject to a brief replication lag of $< 1\text{ second}$).
  • Once US-West comes back online, Cassandra's Hinted Handoffs and background repair loops automatically synchronize any updates missed during the outage.

2. Cross-Region WAN Link Isolation (Split-Brain)

If the WAN connection between datacenters is severed but both datacenters remain online, the regions cannot synchronize updates, creating a split-brain scenario.

• Recovery Protocol:
  • Because Cassandra is configured to use LOCAL_QUORUM, both regions continue to accept writes locally, maintaining high availability for users.
  • Once the WAN connection is restored, Cassandra replicates the accumulated state changes asynchronously.
  • The Flink and application layers merge the divergent states deterministically using the built-in OR-Set CRDT logic, ensuring both regions converge to the correct, unified state.

9. Candidate Verbal Script (Interview Guide)

Below is an exhaustive, verbatim transcript showing how a Staff Engineer candidate navigates the multi-region active-active shopping cart design in a system design interview:

Interviewer: "How would you design a global, highly available shopping cart system that can survive a complete regional outage without dropping users' items?"

Candidate: "I will architect a multi-region, active-active shopping cart system optimized for high write availability, utilizing an AP (Available/Partition-Tolerant) consistency model.

To handle traffic globally, I will deploy stateless microservice fleets in multiple regions (e.g., US-West and EU-Central) behind a DNS Anycast load balancer to route users to the nearest active datacenter.

For persistence, I will use Apache Cassandra configured with NetworkTopologyStrategy to handle masterless, multi-datacenter replication. We will write cart updates using a LOCAL_QUORUM consistency level to guarantee low-latency writes ($< 10\text{ms}$) locally, and let Cassandra replicate updates across regions asynchronously.

To resolve concurrent edit conflicts across regions deterministically, I will implement an Observed-Remove Set (OR-Set) CRDT model in our application layer, tracking additions with unique tags and managing removals using a tombstone set. This ensures that all replicas converge to the correct state once synchronization occurs, avoiding the clock skew issues of Last-Write-Wins models."

Interviewer: "What happens if a user clicks checkout in one region before the latest cart updates have replicated from another region? Won't they buy the wrong items?"

Candidate: "That is a critical edge case. While we use LOCAL_QUORUM for fast, day-to-day cart updates, we cannot tolerate stale data during checkout.

When the user clicks 'Checkout', the API Gateway elevates the read consistency level to QUORUM across data centers. This requires agreement from a majority of global Cassandra nodes (e.g., 4 out of 6 nodes across the US and EU) for the read.

This global quorum read forces Cassandra to resolve any outstanding replication lag on the fly, guaranteeing that the checkout process operates on the most up-to-date cart state."