System Design: Designing Nearby Friends (Real-time Geospatial Streams)

Designing a "Nearby Friends" feature (like Snapchat's Snap Map or Find My) presents a unique high-scale architecture problem. Unlike traditional Yelp-like proximity searches where the coordinates of businesses are static, or Uber-like routing where only active drivers update their positions, "Nearby Friends" requires tracking millions of actively moving users simultaneously and blasting these location changes immediately to their social graph.

1. Requirements & Core Constraints

To build a real-time geographic telemetry platform, we must detail the limits and operational constraints of our system.

Functional Requirements

• Telemetry Broadcasting: Active users must transmit their GPS coordinates (latitude, longitude) regularly while the app is active.
• Dynamic Friend Tracking: Users must see a live map showing the moving locations of their online friends who are within a 5 km radius.
• Proximity Alerts: The system must trigger a notification when a friend enters a 1 km radius of the user.
• Privacy Gating (Ghost Mode): Users must be able to opt-out of sharing their location, use coarse locations (neighborhood-level), or select a "Ghost Mode" that completely hides their map pin.

Non-Functional Requirements

• Low Latency Pipeline: Changes in a user's location should appear on their active friends' screens within 2-3 seconds.
• Extreme Write Capacity: The system must ingest location telemetry updates from millions of active mobile devices without lagging.
• Highly Concurrent Fan-out: Location updates must be pushed efficiently to the social graph of active friends without crushing the network layers.
• High Availability: The core WebSocket and Pub/Sub routing mesh must handle node failovers without dropping existing user telemetry streams.

Back-of-the-Envelope Estimation

• System Demands & Capacity Planning:
  • Daily Active Users (DAU): $100,000,000$ users globally.
  • Concurrent Active Users (Peak hour): Assuming $10%$ of DAUs are online at any given peak hour: $$\text{Concurrent Peak Users} = 10,000,000$$
  • Telemetry Ping Frequency: Active users ping their coordinates once every 10 seconds.
  • Write/Ingress QPS: $$\text{Write QPS} = \frac{10,000,000 \text{ users}}{10 \text{ seconds}} = 1,000,000 \text{ telemetry writes/sec}$$
  • Social Fan-out Load:
    • Assume each active user has 200 friends, of whom $10%$ are currently online and within the search grid: $$\text{Active Friend Recipients} = 20 \text{ friends}$$
    • This creates an outgoing message stream of: $$\text{Fan-out Egress QPS} = 1,000,000 \text{ writes/sec} \times 20 = 20,000,000 \text{ updates/sec!}$$
  • Network Bandwidth:
    • At 100 bytes per incoming WebSocket frame (userId, lat, lng, timestamp): $$\text{Ingress Bandwidth} = 1,000,000 \text{ writes/sec} \times 100 \text{ bytes} = 100 \text{ MB/s}$$
    • At 100 bytes per outgoing friend location push: $$\text{Egress Bandwidth} = 20,000,000 \text{ updates/sec} \times 100 \text{ bytes} = 2.0 \text{ GB/s (16 Gbps)}$$
  • Memory Footprint: Storing a user's latest location in an in-memory geo-index. 100,000,000 users $\times$ 80 bytes (userId, geohash, timestamp) = 8.0 GB (easily fits on a single standard cloud node, replicated for redundancy).

2. API Design & Core Contracts

Because location tracking requires bidirectional, low-latency communication, standard HTTP polling is inefficient. We utilize persistent WebSockets for real-time streaming.

Connection Establishment (WS Handshake)

GET /api/v1/telemetry/ws Upgrades HTTP connection to WebSocket protocol, passing authorization tokens.

Request Headers:

Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Authorization: Bearer <jwt_telemetry_token>

Telemetry Update Frame (Client -> Server)

Sent by the active client device every 10 seconds (or dynamically based on movement velocity).

{
  "event": "client_telemetry",
  "latitude": 37.7749,
  "longitude": -122.4194,
  "accuracy_meters": 10.0,
  "timestamp": 1782236400
}

Friend Telemetry Push (Server -> Client)

Pushed by the WebSocket gateway server to online friends subscribed to the user's grid.

{
  "event": "friend_telemetry",
  "friend_id": "usr_7389a1b0",
  "latitude": 37.7758,
  "longitude": -122.4182,
  "accuracy_meters": 12.0,
  "timestamp": 1782236402
}

3. High-Level Design (HLD)

The architecture is divided into two distinct processing components: the Ingress WebSocket Fleet (which ingests live GPS updates and tracks active sessions) and the Geospatial Pub/Sub Mesh (which shards geographic cells and routes location updates directly to subscribed friends).

graph TD
    Client[Traveler Mobile Client] -->|1. WS Telemetry Handshake| LB[WS Load Balancer]
    LB --> WSFleet[WebSocket Gateway Fleet]
    
    %% Ingress & Telemetry Path
    WSFleet -->|2. Push Location Ping| RedisGeo[(Redis Geospatial Cache)]
    WSFleet -->|3. Route Update| GeohashPubSub[Redis Pub/Sub Geohash Clusters]
    
    %% Relationships & Privacy Gating
    WSFleet -->|4. Verify Friendships & Ghost Mode| AuthzService[Privacy & Relationship Service]
    AuthzService -->|Read Permissions| RelationalDB[(PostgreSQL Primary)]
    
    %% Messaging & Notification
    GeohashPubSub -->|5. Friend Channel Broadcast| WSFleet
    WSFleet -->|6. Push Live Map Marker| DirectFriend[Active Online Friend Client]
    
    %% Analytics & Backup Logs
    RedisGeo -->|CDC Stream| Kafka[Kafka Telemetry Logging]
    Kafka -->|Store Analytics| ClickHouse[(ClickHouse Time-Series DB)]

Flow Lifecycle:

1. Connecting: The user establishes a persistent WebSocket connection through the Load Balancer to a node in the WebSocket Gateway Fleet. This node tracks the user's active session and connection state.
2. Ingressing: The client periodically pushes location coordinates. The receiving WS Gateway writes the coordinates to the Redis Geospatial Cache using GEOADD and pushes the telemetry event into the Redis Pub/Sub Cluster mapped to the user's current Geohash cell.
3. Routing (Fan-out): The Redis Pub/Sub cluster broadcasts the update. Active friends subscribed to the target Geohash cell receive the event through their respective WebSocket Gateway nodes.
4. Filtering (Privacy check): The WebSocket Gateways query the Privacy Service in memory to verify friendship states and ensure the user is not in "Ghost Mode" before pushing updates to target clients.

4. Low-Level Design (LLD) & Data Models

Database Selection Rationale

• Location Cache & Subscription Routing (Redis): Redis stores geographical locations in a Sorted Set (using Geohash values as scores). This supports $O(\log N)$ spatial index updates (GEOADD) and circle range queries (GEORADIUS). Redis Pub/Sub provides low-latency brokerless message routing.
• Relational Social Graph (PostgreSQL): Storing core user profiles, friend networks, and privacy configurations requires strong schema enforcement and compound indices.
• Historical Analysis (ClickHouse): A column-oriented database optimized for high-volume geographical logs is perfect for offline data analysis (e.g., matching travel history or hot locations).

SQL DDL Database Schemas

Friendship Relationships

CREATE TABLE friendships (
    user_id VARCHAR(64) NOT NULL,
    friend_id VARCHAR(64) NOT NULL,
    status VARCHAR(32) NOT NULL DEFAULT 'ACTIVE',
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_id, friend_id)
);

CREATE INDEX idx_friendships_lookup ON friendships (user_id) WHERE status = 'ACTIVE';

User Privacy Configurations

CREATE TABLE user_privacy (
    user_id VARCHAR(64) PRIMARY KEY,
    sharing_mode VARCHAR(32) NOT NULL DEFAULT 'PUBLIC', -- 'PUBLIC', 'COARSE', 'GHOST'
    coarse_radius_meters INT NOT NULL DEFAULT 500,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

Redis Data Mapping Layout

• Location Cache Structure (Sorted Set): Key: locations:region_us_west Score: Geohash spatial index Member: user_id
• Redis Pub/Sub Geohash Routing Channels: We convert coordinates to a Geohash string of length 5 (representing a $4.9\text{km} \times 4.9\text{km}$ grid cell). Channel: geohash:9q8yy

5. Scaling Challenges & High Write Concurrency

Ingesting 1M updates/s and pushing them to millions of active clients creates several core bottlenecks:

Mitigating the Fan-out Blast

If we broad-cast a user's location to all 200 of their friends, we create massive, unnecessary network load, since 90% of those friends are either offline or miles away.

• Active Subscription Windows: A client app only subscribes to the Geohash cell the traveler is currently looking at, plus the 8 surrounding cells. The WebSocket Gateway maps these 9 active cells for the traveler.
• Geohash Sharding: Instead of broadcasting to individual friend IDs, updates are published to the Geohash Cell Channel (e.g., geohash:9q8yy). Since active users in the same neighborhood subscribe to the same cell, the broker routes location updates in a single, localized broadcast.

graph TD
    subgraph Geohash Grid Shards
        Cell1[geohash:9q8yy - Target Cell]
        Cell2[9q8yx - West Neighbor]
        Cell3[9q8yz - East Neighbor]
    end

    Traveler[Traveler Ping] -->|GEOADD & PUBLISH| Cell1
    ActiveFriend[Active Friend Client] -->|WS Subscribe| Cell1
    ActiveFriend -->|WS Subscribe| Cell2
    ActiveFriend -->|WS Subscribe| Cell3

6. Technical Trade-offs & Consistency Models

Designing real-time spatial pipelines requires balancing battery limits, network bandwidth, and data accuracy:

Tradeoff A: WebSocket Streams vs. HTTP Long-Polling

• HTTP Long-Polling:
  • Pros: Simple to implement and scale behind standard HTTP load balancers.
  • Cons: High header overhead (HTTP headers sent with every 10s ping), creating massive network load and draining mobile batteries.
• WebSockets:
  • Pros: Bidirectional, TCP-level connection. Minimal header overhead (2 bytes per frame), enabling sub-second latencies and massive battery savings.
  • Cons: Stateful connection management. If a server node crashes, thousands of connections are lost simultaneously, creating a thundering herd retry storm.

Tradeoff B: Strong Consistency vs. Eventual Consistency for Friend Maps

In Nearby Friends, showing a friend's location exactly where they are at this millisecond is unnecessary. A lag of 5-10 seconds is completely acceptable to the traveler. We choose Eventual Consistency. We store location records in-memory with a short TTL (10 minutes) and use asynchronous Redis Pub/Sub streams to route updates. If a packet is lost, the next ping 10 seconds later corrects the state, making expensive transactional database checkpoints unnecessary.

7. Resilience & Failure Scenarios

Operating stateful WebSocket fleets at scale requires robust recovery protocols:

Scenario A: The WebSocket Thundering Herd Storm

When a WebSocket server node crashes, 50,000 active mobile clients lose connection simultaneously. If they all attempt to reconnect and resubscribe immediately, they will overwhelm the API Gateways.

• The Mitigation: Exponential Jittered Reconnects. The client application uses an exponential backoff reconnect loop with added random jitter (e.g., $t = 2^x + \text{rand}(0, 5)$ seconds) to smoothly spread the reconnection load over several minutes.
• Graceful WS Handshake Rate Limiting: The API Gateway uses token bucket limits to drop excess handshake requests, returning HTTP 429 during recovery peaks.

Scenario B: Mobile Battery and Bandwidth Drainage

Sending GPS telemetry pings every 10 seconds when a user is sitting at home or sleeping wastes mobile battery and server bandwidth.

• The Mitigation: Adaptive Telemetry Engine. The client app monitors the device's accelerometer. If the device is stationary, the ping frequency is reduced to once every 10 minutes. If the device begins moving (e.g., walking or driving), the ping frequency dynamically scales up to once every 10 seconds to maintain high map accuracy.

8. Staff Engineer Perspective (Deep-Dive Callouts)

9. Candidate Verbal Script (Interview Guide)

Interviewer: "How would you design a real-time Nearby Friends system that scales to 10M active concurrent users without draining their mobile batteries?"

Candidate: "First, we must avoid high-frequency HTTP polling. I would establish persistent, bidirectional WebSocket connections between the mobile clients and our WebSocket Gateway Fleet. This reduces header overhead and allows us to push location updates instantly.

To protect the device's battery, we implement Adaptive Telemetry. The mobile application monitors the device's hardware accelerometer. If the user is stationary (e.g., sitting at a desk or sleeping), the client enters a low-power mode and reduces GPS pings to once every 10 minutes. When the accelerometer detects motion, the app increases the ping rate dynamically—up to once every 10 seconds when driving—ensuring high map accuracy only when the user is moving.

At the server layer, we avoid expensive database writes. GPS coordinates are written to a Redis Geospatial cache with a 10-minute TTL and routed through sharded Redis Pub/Sub channels mapped to the user's current Geohash cell. This keeps the entire telemetry loop in-memory, achieving sub-second latencies."

Interviewer: "How do you handle the 'Fan-out' bottleneck if a user has 500 active friends who all need to see their map updates?"

Candidate: "A naive approach where the server queries the database for all 500 friends and sends 500 individual network messages for every single movement will crash the network gateways.

Instead, we use a Geohash-Based Pub/Sub model. The world is divided into discrete Geohash grid cells of length 5 (about $4.9\text{km} \times 4.9\text{km}$). When a traveler moves, the WebSocket Gateway publishes their new coordinates to the Redis Pub/Sub channel representing that specific Geohash cell (e.g., geohash:9q8yy).

Travelers who are actively viewing the map only subscribe to the channel of the Geohash cell they are currently viewing, plus the 8 surrounding neighbor cells. This reduces subscription matching to a simple $O(1)$ channel subscription. The WebSocket Gateway then acts as a filter, querying our in-memory privacy cache to confirm friendship and ensure 'Ghost Mode' is respected before forwarding the coordinate frame to the client."