How to Handle Duplicate Messages and Message Ordering in Kafka

Dealing with the challenges faced when using Apache Kafka

Image by author

Event-driven architecture is one of the ways to implement interprocess communication between some services in a microservice architecture. A message-based application typically uses a message broker which acts as an intermediate layer between the services.

Apache Kafka is one of the tools which allows implementing a broker-based messaging approach. This tool resolves a lot of issues on its own from the box, but as a developer, we do come across challenges that should be taken into account. In the next few sections, we’ll consider some of them and have a look at how to resolve them.

1. Message Ordering

Some applications are pretty sensitive to the order of messages being processed by consumers. Let’s imagine that the user creates some Order, modifies it, and finally cancels. This means, that the Producer sends to Message Broker the following messages with commands: ORDER_CREATE, ORDER_MODIFY, ORDER_CANCEL.

In Kafka, we can create a special topic Order which will get all messages (commands) that will process any operation related to the orders. If our application has only one Consumer and one topic with one partition, you will never…