Worker Queue Processing

Worker queue processing is a messaging pattern where a Producer places work items into a Message Broker queue and one or more Worker processes consume and process those items asynchronously, with explicit acknowledgement ensuring at-least-once delivery.

What the diagram shows

This sequence diagram involves three participants: Producer (the application that creates work), Message Broker (e.g., RabbitMQ, SQS, Redis Streams), and Worker (the consumer process).

The flow illustrates:

1. Publish: the producer serializes a message (job payload) and publishes it to a named queue on the broker. The broker persists the message and acknowledges receipt. 2. Consume: the worker subscribes to the queue or polls it. The broker delivers a message to the worker and marks it as "in-flight" (invisible to other consumers for a visibility timeout period). 3. Process: the worker executes the job. If processing succeeds, the worker sends an explicit ACK (acknowledgement) to the broker, which permanently removes the message. 4. Negative acknowledgement: if processing fails, the worker sends a NACK or simply lets the visibility timeout expire. The broker returns the message to the queue for redelivery. 5. Competing consumers: multiple worker instances can consume from the same queue concurrently — the broker ensures each message is delivered to only one worker at a time (competing consumer pattern).

Why this matters

The explicit ACK/NACK protocol is what makes message queues reliable. Unlike a simple in-memory queue, a message is not lost if a worker crashes mid-processing — the visibility timeout causes it to reappear in the queue for another worker to claim.

For the full job lifecycle including retry budgets and dead letters, see Background Job Processing. For event streaming (as opposed to task queue) semantics, see Kafka Producer Consumer Flow. For time-triggered jobs, see Cron Job Scheduler.