Exactly Once Delivery: Mermaid Flowchart

About Source

Exactly-once delivery is the strongest message delivery semantic, guaranteeing that each message is processed precisely one time — neither lost (at-most-once) nor processed multiple times (at-least-once) — even in the presence of producer retries, consumer failures, and network partitions.

True exactly-once delivery requires coordination at both the producer and consumer levels, and is significantly harder to achieve than the two weaker semantics it supersedes.

At-most-once delivery is the simplest: the producer sends without retrying, so messages can be lost on network failure but are never duplicated. At-least-once delivery adds producer retries and consumer-side offset commits after processing, ensuring no message is lost but allowing duplicates if a crash occurs between processing and committing. Exactly-once combines idempotent production with transactional commit to close the duplication window.

In Kafka, exactly-once semantics (EOS) are implemented through two mechanisms working together. On the producer side, enable.idempotence=true gives each producer a unique PID and assigns monotonically increasing sequence numbers to each message per partition. The broker deduplicates retried messages using these sequence numbers. On the consumer-producer side (for stream processing), Kafka transactions (transactional.id) allow an atomic operation: read from source partitions, process, write to output partitions, and commit offsets — all as a single atomic unit. Either all succeed or all are rolled back.

The cost is latency: transactional commits require additional broker round-trips. For most applications, Idempotent Consumer design achieves the same practical outcome — deduplicated side effects — without the Kafka transaction overhead, by tracking processed Message Deduplication keys at the application level.

Frequently asked questions

Exactly-once delivery is the strongest message delivery semantic, guaranteeing that each message is processed precisely one time — not lost (at-most-once) and not processed multiple times (at-least-once). It requires coordinated mechanisms at both the producer and consumer levels to close the duplication window that exists in at-least-once systems.

In Kafka, exactly-once semantics (EOS) combine two mechanisms. Idempotent producers (`enable.idempotence=true`) assign sequence numbers per partition; the broker deduplicates retried messages using these numbers. Kafka transactions (`transactional.id`) extend this to stream processing by making read-process-write-commit-offset a single atomic operation — either the entire unit succeeds or it is rolled back.

Use exactly-once semantics when duplicate processing has unacceptable consequences — financial transactions, inventory deductions, or any operation that cannot be safely rolled back or deduplicated at the application level. For most workloads, at-least-once delivery combined with an idempotent consumer achieves the same practical result with lower latency and simpler configuration.

The most common mistake is assuming that enabling `enable.idempotence=true` on the producer alone gives end-to-end exactly-once semantics — it only deduplicates at the producer-to-broker boundary. True EOS also requires transactional consumers. Another mistake is using exactly-once semantics when idempotent consumer design would suffice, unnecessarily paying the latency cost of transactional commits.

mermaid

flowchart TD
    subgraph AtMostOnce[At-Most-Once]
        P1[Producer] -->|send, no retry| B1[Broker]
        B1 -->|maybe lost| C1[Consumer]
    end

    subgraph AtLeastOnce[At-Least-Once]
        P2[Producer\nretries on timeout] -->|send + retry| B2[Broker]
        B2 -->|possible duplicate| C2[Consumer]
        C2 -->|commit offset after ack| B2
    end

    subgraph ExactlyOnce[Exactly-Once - Kafka EOS]
        P3[Idempotent Producer\nenable.idempotence=true\nPID + sequence number] -->|send with seq=N| B3[Broker]
        B3 -->|deduplicate retries by seq| B3
        B3 --> TX[Kafka Transaction\ntransactional.id]
        TX -->|atomic: read + process + write + commit offset| C3[Transactional Consumer]
        C3 -->|commit or abort\nall-or-nothing| TX
    end

    style AtMostOnce fill:#faa,color:#000
    style AtLeastOnce fill:#ffa,color:#000
    style ExactlyOnce fill:#afa,color:#000