A message queue lets your system absorb a sudden spike of 100,000 events without breaking — processing them steadily over minutes instead of drowning in milliseconds.
What's actually happening here?
A message queue is a durable buffer between a producer (something that creates work) and a consumer (something that processes work). The producer puts messages into the queue as fast as it needs to. The consumer pulls from the queue at whatever rate it can handle. The queue absorbs the difference. If the producer suddenly sends 50,000 messages in one second but the consumer can only process 5,000 per second, the queue holds the backlog and the consumer works through it over the next 10 seconds. Without a queue, those 50,000 simultaneous requests would hit your processing logic directly and either overwhelm it or get dropped.
The problem this solves
Synchronous systems are brittle at scale. When a user uploads a video to YouTube, compressing it to 12 different resolutions takes minutes — far too long to make the user wait for an HTTP response. When Zomato receives 10,000 orders in a minute during dinner peak, it cannot process payments, notify restaurants, assign riders, and send SMS confirmations all synchronously in one request. Message queues decouple the act of receiving work from the act of doing work. The user gets an immediate response. The heavy work happens asynchronously, at the pace the system can sustain.
How it really works (step by step)
The lifecycle of one message:
Producer publishes a message — the Order Service calls
queue.publish("order.created", {order_id: 4821, ...}). This takes ~1ms and returns immediately. The producer's job is done.Queue persists the message — written to disk before acknowledging. If the queue server crashes right now, the message survives. This durability guarantee is what makes queues trustworthy.
Consumer polls or receives a push — a worker process either polls the queue (
SQSlong-polling) or receives a push (Kafkapartition assignment). The message is delivered to one consumer in the group — not all of them.Consumer processes the message — runs the business logic: charge payment, call Stripe, update database. This might take 200ms or 30 seconds — the queue doesn't care.
Consumer acknowledges — sends an explicit
ackback to the queue only after successful processing. This is the critical step. Until the ack arrives, the queue considers the message unprocessed.Queue deletes the message — only after ack. If the consumer crashes before acking, the message becomes visible again after a timeout (visibility timeout in SQS, consumer group rebalance in Kafka) and another consumer picks it up. No work is lost.
Dead letter queue catches poison pills — if a message fails processing 3 times (configurable), it moves to a Dead Letter Queue for inspection. It doesn't block the queue or consume retry budget infinitely.
The part most tutorials skip
At-least-once delivery is the default — and your consumer must be idempotent. Every production queue guarantees at-least-once delivery, not exactly-once. A message can be delivered twice if the consumer processes it successfully but crashes before sending the ack. Your consumer will see duplicate messages. This is not a bug — it is a deliberate design decision, because guaranteeing exactly-once delivery requires distributed coordination that costs significant throughput.
The solution is idempotent consumers: processing the same message twice produces the same result as processing it once. For a payment consumer, this means checking whether a payment for order_id: 4821 already exists before charging. For an email consumer, this means checking whether a confirmation was already sent. The idempotency key pattern — storing processed message IDs in Redis with a TTL — makes any consumer idempotent with ~5 lines of code.
Real company doing this right now
Shopify processes over 100 million jobs per day through a combination of Sidekiq (Redis-backed job queues) and Kafka. When a merchant makes a sale, a cascade of jobs fires: inventory update, analytics event, webhook delivery to the merchant's third-party integrations, email receipt, fraud scoring. Each is an independent message on an independent queue. The critical insight Shopify published: they size each consumer pool independently based on that job type's throughput needs and failure characteristics. Webhook delivery consumers are small and many — webhooks fail often and need aggressive retries. Inventory consumers are large and few — inventory updates must be strictly ordered per product. One queue infrastructure, completely different consumer configurations per job type.
What breaks at scale?
Consumer lag is the metric that predicts outages before they happen. Consumer lag is the difference between the newest message in the queue and the message the consumer is currently processing. Lag of 0 means consumers are keeping up. Lag of 100,000 messages means your consumers are falling behind and the queue is growing. Left unmonitored, lag compounds — a consumer that falls 10 minutes behind during a spike may never catch up if the underlying cause isn't fixed. At Uber, consumer lag on their location update pipeline growing beyond a threshold automatically triggers provisioning of additional consumer instances via Kubernetes HPA — the system heals itself before an engineer notices.
The "aha" moment
A queue is not a database — it is a conveyor belt. Data on a conveyor belt is meant to move, be processed, and disappear. Designing a queue as long-term storage, or reading the same message multiple times for different purposes, means you need Kafka's log model — not a traditional queue.
Your practical takeaway
Set a Dead Letter Queue on every queue from day one — without a DLQ, a message that causes your consumer to crash will be retried indefinitely, blocking processing for everything behind it. A DLQ catches it after 3 failures, surfaces it for debugging, and lets the rest of the queue continue.
Make every consumer idempotent before you care about performance — add a Redis check for processed message IDs (TTL matching your queue's retention period) as the first thing every consumer does. This costs ~0.1ms and eliminates an entire class of double-processing bugs that are nearly impossible to debug in production.
Monitor consumer lag as a primary metric, not an afterthought — set an alert when lag exceeds 2× your normal peak. A growing lag is always a symptom of something — insufficient consumers, slow database, downstream service degraded. Catching it at 2× gives you time to fix it before users notice.
Lesson 14 · Stage 4 — Scalability Patterns · System Design Made Easy