Managing Message Reliability
1. At-Least-Once Delivery Pattern
| Aspect | Detail |
|---|---|
| Guarantee | Message delivered ≥1 time; possibly duplicates |
| Mechanism | Producer retries on broker failure; consumer acks after processing |
| Required | Idempotent consumer |
| Default In | Kafka, RabbitMQ, SQS |
2. At-Most-Once Delivery Pattern
| Aspect | Detail |
|---|---|
| Guarantee | Message delivered ≤1 time; may be lost |
| Mechanism | Producer fire-and-forget; consumer auto-ack before processing |
| Use Case | Metrics, telemetry where loss tolerable |
3. Exactly-Once Delivery Pattern
| Aspect | Detail |
|---|---|
| Guarantee | Each message processed exactly once (effectively) |
| Achieved Via | At-least-once + idempotent consumer (most common) |
| Native Support | Kafka transactions (EOS), Pulsar, Flink checkpointing |
| Cost | Higher latency, more coordination |
Note: True end-to-end exactly-once across systems is impossible (Two Generals). Implement effectively-once via idempotency.
4. Idempotent Consumer Pattern
| Strategy | Detail |
|---|---|
| Dedup Table | Record event ID with side-effect in same tx |
| Natural Idempotency | Operation safe to repeat (e.g., SET x = 5) |
| Conditional Update | UPDATE WHERE version = expected |
| Upsert | INSERT ... ON CONFLICT DO NOTHING |
Example: Idempotent Insert
INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;
-- If 0 rows affected → duplicate; skip side effect5. Message Acknowledgment Pattern
| Mode | Detail |
|---|---|
| Auto-Ack | Broker marks delivered on send; risk of loss |
| Manual Ack | Consumer explicitly acks after success |
| Negative Ack (NACK) | Reject + requeue or send to DLQ |
| Batch Ack | Ack range of offsets (Kafka commit) |
6. Dead Letter Queue Pattern
| Aspect | Detail |
|---|---|
| Purpose | Park messages that fail repeatedly |
| Trigger | Max retries exceeded, deserialization fail, validation fail |
| Action | Alert, manual inspection, replay after fix |
| Retention | Longer than main queue (days/weeks) |
Example: RabbitMQ DLX Configuration
Map<String, Object> args = Map.of(
"x-dead-letter-exchange", "orders.dlx",
"x-dead-letter-routing-key", "orders.failed",
"x-message-ttl", 60000);
channel.queueDeclare("orders", true, false, false, args);7. Poison Message Handling Pattern
| Step | Action |
|---|---|
| 1. Detect | Failure on every retry → poison |
| 2. Quarantine | Move to DLQ; do not block other messages |
| 3. Alert | Notify on-call with payload + error |
| 4. Diagnose | Inspect payload; fix code or data |
| 5. Replay | Re-publish to main queue once safe |
8. Message Deduplication Pattern
| Method | Detail |
|---|---|
| Producer-Side | Idempotent producer (Kafka enable.idempotence=true) |
| Broker-Side | SQS FIFO dedup window (5 min); Kafka EOS |
| Consumer-Side | Inbox/dedup table |
| End-to-End | Combine producer + consumer dedup |
9. Inbox Pattern
| Aspect | Detail |
|---|---|
| Purpose | Atomically dedupe + persist incoming events |
| Mechanism | Insert event ID into inbox table in same tx as side-effect |
| Duplicate | Insert fails (PK conflict) → skip processing |
| Cleanup | Periodic delete of old inbox rows |
Example: Inbox Atomic Processing
BEGIN;
INSERT INTO inbox(event_id, received_at) VALUES ($1, now()); -- fails on duplicate
UPDATE accounts SET balance = balance + $2 WHERE id = $3;
COMMIT;10. Priority Queue Pattern
| Approach | Detail |
|---|---|
| Multiple Queues | Separate queue per priority; consumers prefer high |
| Native Priority | RabbitMQ x-max-priority; ActiveMQ priority |
| Fairness | Use weighted polling to avoid starving low priority |
| Note on Kafka | No native priority; use multiple topics or partitioning |