Working with Message Queues
1. Understanding Message Queue Patterns
| Pattern | Description |
|---|---|
| Point-to-point | One producer, one consumer per msg (queue) |
| Pub/Sub | One producer, many subscribers (topic/fanout) |
| Competing consumers | Many workers share one queue |
| Request-reply | Reply queue + correlation id |
| Routing | Topic/header-based routing (RabbitMQ exchanges) |
2. Implementing Producer-Consumer Pattern
Example: Kafka producer/consumer
// Producer
KafkaProducer<String, String> p = new KafkaProducer<>(props);
p.send(new ProducerRecord<>("orders", order.id, json));
// Consumer
KafkaConsumer<String, String> c = new KafkaConsumer<>(props);
c.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> recs = c.poll(Duration.ofMillis(500));
for (var r : recs) handle(r);
c.commitSync();
}3. Implementing Message Acknowledgments
| Mode | Semantics |
|---|---|
| Auto-ack | Acked on receive; risk of loss |
| Manual ack after process | At-least-once |
| Negative ack (nack) | Reject + requeue / DLQ |
| Cumulative ack | Single ack covers offset range (Kafka) |
4. Implementing Exactly-Once Semantics
| Mechanism | System |
|---|---|
| Idempotent producer + transactions | Kafka EOS (read-process-write) |
| Dedup at consumer (idempotency key) | App layer |
| Two-phase commit with sink | Flink, Beam |
| Outbox pattern | DB tx writes msg + state |
Warning: Pure exactly-once across boundaries is impossible (Two Generals); achieve "effectively once" via idempotency + dedup.
5. Handling Message Ordering Guarantees
| Guarantee | System |
|---|---|
| Per-partition FIFO | Kafka, Kinesis |
| Per-queue FIFO | SQS FIFO, Azure Service Bus sessions |
| Group ordering | RabbitMQ single consumer; partition by key |
| Global ordering | Single partition (low throughput) |
6. Implementing Dead Letter Queues
| Trigger | Action |
|---|---|
| Max retries exceeded | Move to DLQ |
| TTL expired | Move to DLQ |
| Queue length limit | Drop or DLQ |
| Operator action | Inspect, replay, or discard |
| Alerts | Monitor DLQ depth as SLI |
7. Implementing Message Expiration (TTL)
| Scope | Mechanism |
|---|---|
| Per-message TTL | Header (RabbitMQ expiration) |
| Per-queue TTL | Default for all in queue |
| Topic retention | Kafka retention.ms |
| SQS | Visibility timeout + 14-day max |
8. Implementing Priority Queues
| Approach | Detail |
|---|---|
| Priority field | RabbitMQ x-max-priority |
| Multiple queues per level | Workers poll high → low |
| Weighted fair queue | Round-robin with weights |
9. Handling Poison Messages
| Step | Action |
|---|---|
| Detect | Counter on retries; consistent failure |
| Quarantine | Move to DLQ after threshold |
| Alert | Engineer investigates payload |
| Replay | After fix, re-enqueue from DLQ |
10. Implementing Message Batching
| Knob | Effect |
|---|---|
| Batch size | Bigger = better throughput, higher latency |
| Linger time | Wait to fill batch (Kafka linger.ms) |
| Compression | gzip, snappy, lz4, zstd — applied per batch |
| Trade-off | Throughput ↑, end-to-end latency ↑ |
11. Understanding Message Broker Architectures
| Broker | Architecture | Best For |
|---|---|---|
| Kafka | Distributed log, partitioned | High throughput, replay |
| RabbitMQ | Smart broker, routing | Complex routing, low-mid scale |
| NATS / JetStream | Lightweight pub/sub | Microservices, IoT |
| Pulsar | Segmented log + tiered storage | Multi-tenant, geo-replication |
| SQS / Pub/Sub / EventBridge | Managed, serverless | Cloud-native, no ops |
| Redis Streams | In-memory log | Low-latency, smaller scale |