Designing Idempotency Patterns
1. Understanding Idempotency Concepts
| Term | Meaning |
|---|---|
| Idempotent | f(f(x)) == f(x); safe to retry |
| Naturally idempotent | PUT /resource/123, DELETE |
| Made idempotent | POST + idempotency key |
| Effectively-once | At-least-once delivery + idempotent processing |
2. Designing Idempotent APIs
| HTTP Verb | Idempotent? | Notes |
|---|---|---|
| GET | Yes | Safe + idempotent |
| PUT | Yes | Replace |
| DELETE | Yes | Subsequent returns 404 or 204 |
| POST | No (default) | Add Idempotency-Key header |
| PATCH | Depends | Idempotent if absolute, not if delta |
3. Designing Request Deduplication
Example: Idempotency key (Stripe-style)
POST /v1/charges
Idempotency-Key: 9b2a3c1e-7d22-4d4a-8d2e-...
Server:
if (cache.has(key)) return cache.get(key); // same response
result = doWork(req);
cache.put(key, result, ttl=24h);
return result;| Element | Detail |
|---|---|
| Key source | Client-generated UUID per logical request |
| Storage | Redis or DB with TTL (24h typical) |
| Scope | Per endpoint + tenant |
| Concurrent same-key | Lock or 409 Conflict |
4. Implementing Idempotency in Message Processing
| Technique | Detail |
|---|---|
| Dedup by message ID | Track processed IDs in store |
| Idempotent op | UPSERT, conditional update |
| Outbox + inbox | Inbox table dedups before applying |
| Kafka idempotent producer | enable.idempotence=true |
5. Designing At-Least-Once with Idempotency
| Component | Role |
|---|---|
| Producer | Retry until ack |
| Consumer | Process + dedup before commit offset |
| Effective semantics | Exactly-once outcome |
6. Designing Exactly-Once Semantics
| System | Mechanism |
|---|---|
| Kafka EOS | Idempotent producer + transactions |
| Flink checkpoints | Two-phase commit sink |
| DB outbox | Write event + state in same txn |
| Pulsar | Deduplication enabled at broker |
Warning: True end-to-end exactly-once requires coordination at every hop. Most systems achieve "effectively once" via at-least-once + idempotency.
7. Designing Idempotent Database Operations
| Op | Idempotent Form |
|---|---|
| Insert | INSERT ... ON CONFLICT DO NOTHING |
| Upsert | INSERT ... ON CONFLICT DO UPDATE |
| Increment | Use idempotency key + processed table |
| Delete | WHERE id=? (re-run is no-op) |
8. Designing Idempotent Payment Processing
Idempotent Payment Flow
- Client generates idempotency_key (UUID) per checkout attempt
- Server stores key + status (pending) in DB
- Call PSP with idempotency_key (PSPs like Stripe accept this)
- Persist PSP txn id; mark complete
- Retries with same key return stored response
9. Designing Retry-Safe Operations
| Practice | Detail |
|---|---|
| Make ops idempotent | Default for safe retries |
| Exponential backoff + jitter | Avoid thundering herd |
| Bounded retries | Cap attempts; circuit breaker |
| Distinguish errors | Retry transient; surface permanent |
10. Designing Idempotency Token Storage
| Store | Trade-off |
|---|---|
| Redis | Fast; needs persistence/replication for safety |
| Relational DB | Durable + transactional with the operation |
| DynamoDB | Conditional writes for atomic dedup |
| TTL | 24h–7d typical; longer for financial |
| Schema | (key, request_hash, response, status, expires_at) |