Abstract Algorithms

kafka

7 articles across 5 sub-topics

Change Feed vs Change Stream: CDC Internals, Reliability, and When to Avoid Each

In the summer of 2023, the platform team at a fast-growing e-commerce company was handling 100,000 orders per day across three microservices: Order Service, Inventory Service, and Billing Service. All three needed to react to the same database mutati...

Apr 17, 2026•46 min read

How CDC Works Across Databases: PostgreSQL, MySQL, MongoDB, and Beyond

A data engineering team at a fintech company built what they believed was a robust Change Data Capture pipeline: three source databases (PostgreSQL, MongoDB, and Cassandra), Debezium connectors wired to Kafka, and a downstream data warehouse receivin...

Apr 5, 2026•33 min read

Architecture(2)

Kappa Architecture: Streaming-First Data Pipelines

TLDR: Kappa architecture replaces Lambda's batch + speed dual codebases with a single streaming pipeline backed by a replayable Kafka log. Reprocessing becomes replaying from offset 0. One codebase, no drift. TLDR: Kappa is the right call when your t...

Mar 28, 2026•20 min read

System Design HLD Example: E-Commerce Platform (Amazon)

TLDR: A large-scale e-commerce platform separates catalog, cart, inventory, orders, and payments into independent microservices. The core architectural challenge is Inventory Correctness during flash sales—solved with a two-phase reservation pattern:...

Mar 28, 2026•13 min read

#Apache-spark(1)

Kafka and Spark Structured Streaming: Building a Production Pipeline

📖 The 500K-Event Problem: When a Naive Kafka Consumer Falls Apart An analytics platform at a mid-sized fintech company needs to process 500,000 payment events per second from a Kafka cluster. The team starts with a straightforward approach: a hand-r...

Apr 19, 2026•22 min read

Architecture Patterns(1)

The Dual Write Problem: Why Two Writes Always Fail Eventually — and How to Fix It

TLDR: Any service that writes to a database and publishes a message in the same logical operation has a dual write problem. try/catch retries don't fix it — they turn failures into duplicates. The Transactional Outbox pattern co-writes business data ...

Apr 5, 2026•22 min read

Distributed Systems(1)

How Kafka Works: The Log That Never Forgets

TLDR: Kafka is a distributed event store. Unlike a traditional queue (RabbitMQ) where messages disappear after reading, Kafka stores them in a persistent Log. This allows multiple consumers to read the same data at their own pace, replay history, and...

Mar 9, 2026•13 min read