Topic
Kafka
Learn Kafka as a connected topic across chapters, concepts, simulations, and interview reasoning.
10 Concepts9 Articles3h 33m
Overview
Learn Kafka as a connected topic across chapters, concepts, simulations, and interview reasoning.
How this topic helps
System Design
Architecture
Distributed Systems
Data Engineering
Learning Path in this Topic
Series that contain articles from Kafka. Select a path to filter the article list.
Articles
9 matched articles
Article 1Kafka and Spark Structured Streaming: Building a Production Pipeline📖 The 500K-Event Problem: When a Naive Kafka Consumer Falls Apart
An analytics platform at a mid-sized fintech company needs to process 500,000 payment events per second from a Kafka cluster. The tea23 minArticle 2How Kafka Works: The Log That Never ForgetsTLDR: Kafka is a distributed event store. Unlike a traditional queue (RabbitMQ) where messages disappear after reading, Kafka stores them in a persistent Log. This allows multiple consumers to read th13 min
Article 3Change Feed vs Change Stream: CDC Internals, Reliability, and When to Avoid EachIn the summer of 2023, the platform team at a fast-growing e-commerce company was handling 100,000 orders per day across three microservices: Order Service, Inventory Service, and Billing Service. All49 minArticle 4The Dual Write Problem: Why Two Writes Always Fail Eventually — and How to Fix ItTLDR: Any service that writes to a database and publishes a message in the same logical operation has a dual write problem. try/catch retries don't fix it — they turn failures into duplicates. The Tra23 minArticle 5How CDC Works Across Databases: PostgreSQL, MySQL, MongoDB, and BeyondA data engineering team at a fintech company built what they believed was a robust Change Data Capture pipeline: three source databases (PostgreSQL, MongoDB, and Cassandra), Debezium connectors wired 37 min
Article 6Kappa Architecture: Streaming-First Data PipelinesTLDR: Kappa architecture replaces Lambda's batch + speed dual codebases with a single streaming pipeline backed by a replayable Kafka log. Reprocessing becomes replaying from offset 0. One codebase, n21 minPage 1 of 2