Idempotent Consumer: Mermaid Flowchart

About Source

An idempotent consumer is a message handler designed so that processing the same message multiple times produces exactly the same result as processing it once, enabling safe redelivery and retry without risk of duplicate side effects.

In any at-least-once delivery system — which encompasses Kafka, RabbitMQ, SQS, and most managed queuing services — a message may legitimately arrive more than once. This happens during consumer restarts (the consumer replays from the last committed offset), during network timeouts (the broker re-delivers a message whose ack was lost in transit), or after a crash that occurred between processing and committing. Without idempotent consumers, these duplicates translate into duplicate charges, duplicate emails, or inconsistent aggregate state.

Idempotency must be designed at the business logic level, not just the infrastructure level. There are three primary techniques. First, natural idempotency: some operations are inherently idempotent — setting a field to a specific value (UPDATE accounts SET status='verified') is safe to run twice; incrementing a counter is not. Design your data model to prefer set-based operations where possible.

Second, idempotency keys: attach a unique identifier to every message (a UUID or domain key like payment-{order-id}). Before processing, check whether this key has been seen before in a Message Deduplication store. If yes, skip processing and acknowledge. If no, process, then record the key.

Third, conditional writes: use database-level constraints such as INSERT ... ON CONFLICT DO NOTHING or optimistic locking (WHERE version = N) so that duplicate writes fail silently rather than creating duplicate records.

Idempotent consumer design is the application-layer complement to Exactly Once Delivery infrastructure semantics — together they close the gap between at-least-once delivery and true exactly-once behavior.

Frequently asked questions

An idempotent consumer is a message handler designed so that processing the same message multiple times produces exactly the same outcome as processing it once. In at-least-once delivery systems — Kafka, RabbitMQ, SQS, and most managed queuing services — messages can legitimately arrive more than once, and idempotency is the application-level defence against duplicate side effects.

The three primary techniques are: natural idempotency (designing operations to be inherently safe to repeat, such as set-based database writes), idempotency keys (checking a deduplication store before processing and skipping if the key has been seen), and conditional writes (using database constraints like `INSERT ... ON CONFLICT DO NOTHING` so duplicate writes fail silently rather than corrupting data).

Any consumer operating in an at-least-once delivery environment should be designed as idempotent. This is essentially all production messaging systems — Kafka, SQS, RabbitMQ, and similar all have scenarios where a message is delivered more than once. The pattern is especially critical when the side effects are irreversible: charging a payment, sending an email, or decrementing inventory.

The most common mistake is assuming that idempotency is only needed for retried messages. Consumer restarts in Kafka replay from the last committed offset, and the gap between processing and committing an offset is a window for duplicates even without explicit retries. Another mistake is using an idempotency store with a TTL that is too short, allowing late redeliveries to slip through as false new messages.

mermaid

flowchart TD
    MQ[Message Queue] -->|deliver message\nevent_id=uuid-789| C[Consumer]

    C -->|extract event_id| CHK{Check\nProcessed Store}

    CHK -->|event_id NOT found\nnew message| PROC[Process Message\nExecute business logic]
    CHK -->|event_id FOUND\nduplicate delivery| SKIP[Skip — already processed\nno side effects]

    PROC --> DB[(Write to Database\nconditional INSERT\nON CONFLICT DO NOTHING)]
    DB -->|success or conflict\nboth safe| MARK[Mark event_id\nas processed]
    MARK --> ACK[Acknowledge message]
    SKIP --> ACK

    subgraph ProcessedStore[Processed Event Store - Redis]
        CHK
        MARK
        TTL2[TTL: 48 hours per key]
    end

    style SKIP fill:#fa0,color:#000
    style ACK fill:#4a4,color:#fff