HLD: Kafka Outbox Pattern

Before I tell you about Outbox Pattern, we need to know about the dual write problem. Let's say we have a requirement that when we create an order, we have to store the order details in DB, plus we also have to publish an event, Kafka event. The above code has one major flaw in it. What if DB write succeed and Kafka publish fail? Or what if Kafka publish succeed and DB write fail? Because of this, our system could be in inconsistent state. And generally what I see is that engineer provide answer is two-phase commit where the participants are DB and Kafka. But the thing is Kafka do not support two PC protocol. So, answer is Outbox Pattern. As part of one transaction, we write details into the DB, plus instead of publishing to Kafka, we write into a new table called Outbox table. So, no Kafka publish into this transaction. Now, one approach is we write a poller like scheduler which consistently query this Outbox table, fetch the unpublished event, and push it to Kafka. And one of the major disadvantage of this polling strategy is that even though there is no event, we have to continuously query the Outbox table, which like we are putting a load on DB. Another acceptable solution is CDC, Change Data Capture. So, every DB, like Postgres, maintain a WAL file. And similarly, MySQL maintain a bin log file. So, this file is used for replication. So, whenever there is insert, update, delete happen, it also goes into the WAL file, and the same WAL file is shared to CDC, and CDC read this file and publish the events to Kafka. So, no consistent query to our DB now. If you found it interesting, I have shared in-depth of it with proper setup in event-driven architecture with Kafka playlist. Hope you will find it helpful. Bye.