The Transactional Outbox Pattern

In a microservice, you often need to save data to a database (e.g., Order) and send an event to Kafka (e.g., OrderCreated). If the DB write succeeds but the Kafka send fails, your system is inconsistent. This is the Dual Write Problem.

1. Why 2PC Fails at Scale

The Two-Phase Commit (2PC) protocol provides strong consistency but is slow and prone to blocking. In high-scale systems, we prefer Eventual Consistency via the Outbox Pattern.

2. The Mechanics of the Outbox

Instead of sending to Kafka directly, you write the event to a special OUTBOX table in the same database transaction as your business data.

1. START TRANSACTION
2. INSERT INTO orders (...)
3. INSERT INTO outbox (event_payload, status='PENDING')
4. COMMIT

![Diagram showing the atomic DB write and the secondary process reading the Outbox]

3. Polling vs. CDC (Debezium)

How do you get data from the Outbox table into Kafka?

• Polling: A background worker queries the OUTBOX table every second. Simple, but creates additional database load.
• CDC (Change Data Capture): Using a tool like Debezium to read the database transaction logs (WAL). This is high-performance and has zero impact on the application logic.

4. Ensuring Idempotency

Downstream consumers must be idempotent. Since the Outbox pattern guarantees At-Least-Once delivery, a consumer might receive the same event twice during a network flap.

Next: Kafka Internals: The Storage Layer & Zero-Copy Previous: Java Virtual Threads: High-Concurrency without Complexity

Related Guides

• Beginner: What is a Microservice?
• Related: Saga Pattern for Distributed Transactions
• Advanced: Designing an Idempotent API