Reliable Message Delivery Using Transactional Outbox Pattern
In a distributed environment, communication is the key, and failure in communication is not so uncommon. How do we deliver a message reliably and consistently. Let’s Begin.
The Problem
For almost all transactions happening on our website, we do a fraud check(Your money is safe with us). This fraud check operation can result in one of the three states — ACCEPTED, REJECTED or PENDING_REVIEW. PENDING_REVIEW cases require a human intervention, i.e. someone from operations has to go to their dashboard and do ACCEPT/REJECT, this triggers a callback to our Payments system and has to be propagated aptly to the downstream systems. The solution we are using(a very well-known one in its space) has a limitation that it will only invoke this callback once*. So, if any issues arise while processing this callback, it will result in order failure. Also, We need to be sure that this processing and the propagation(through events) to downstream systems happen consistently and atomically**.
This problem is a perfect use-case for Transactional Outbox Pattern.
What is it?
In this pattern, instead of directly writing to the desired table, we first put an entry in an outbox table, we store the payload we get in callback, as is, and mark it as pending, then a worker(we wrote a cron job) reads these entries in batches, then these batches are processed sequentially.For a record in a batch, we write to our original table first and then try to publish the message. If there’s an error in publishing message, we immediately rollback our write. If everything goes well, we process the next record in the current batch and mark this record as processed. The following diagram illustrates it.
Diagram might look daunting, however, it is fairly simple to implement, and it comes with tonnes of advantages.
Pros
- Before we do any processing, we save the callback as is. This solves our first problem(marked as *) as we can always read the payload in case of exceptions.
- This enables us to replay messages at a later point in time. This is a huge advantage.
- This ensures, that we are delivering our messages consistently and atomically.
Conclusion
Microservice architecture comes with many benefits, however they also create multiple points of failure. Transactional Outbox pattern minimises the chances of failure and ensures the atomicity of the operation at same time.