Kafka Producer Consumer Flow

A Kafka producer-consumer flow describes the end-to-end journey of a message from the application that generates it to the application that processes it, with an Apache Kafka cluster acting as the durable, ordered buffer in between.

The flow begins when a producer publishes a record to a named topic. Before the write is acknowledged, Kafka routes the record to a specific partition within that topic — either by hashing the message key or using a round-robin strategy if no key is set. See Kafka Partitioning for how this assignment works. The partition leader (a broker) writes the record to its local log and then replicates it to follower brokers according to the topic's replication factor.

Once the in-sync replicas (ISR) have persisted the record, Kafka sends an acknowledgment back to the producer. The producer's acks setting controls this: acks=0 means fire-and-forget, acks=1 means the leader alone confirms, and acks=all waits for all ISR members — the safest choice for durability.

On the read side, consumers in a Kafka Consumer Group poll partitions continuously. Each consumer maintains an offset — the position of the last record it has processed in each partition. After handling a batch of records, the consumer commits its offset back to the __consumer_offsets topic. If the consumer crashes and restarts, it resumes from the last committed offset, meaning messages are reprocessed at most once (with enable.auto.commit=false) or potentially duplicated if the commit failed — leading directly to the need for Exactly Once Delivery semantics.

Understanding this flow is foundational for diagnosing lag (when consumers fall behind producers), tuning throughput (batch sizes, linger.ms), and reasoning about failure modes. It also underpins higher-level patterns like Event Streaming Architecture and Stream Processing Pipeline.