Implementing Resilience Patterns
1. Bulkhead Pattern
| Aspect | Detail |
|---|---|
| Inspiration | Ship's compartments — flooding contained |
| Mechanism | Isolate resources (threads, connections) per dependency/tenant |
| Benefit | One slow dependency can't exhaust all threads |
| Tools | Resilience4j Bulkhead, separate connection pools |
2. Bulkhead Pool Configuration
| Param | Typical |
|---|---|
| maxConcurrentCalls | 10-50 per dependency |
| maxWaitDuration | 0 (fail fast) or short (100ms) |
| queueCapacity | 0 or small for thread pool bulkhead |
| type | Semaphore (lighter) or ThreadPool (better isolation) |
3. Retry Pattern
| Aspect | Detail |
|---|---|
| When | Transient failures only (5xx, timeouts, 429) |
| Never Retry | 4xx (validation, auth), non-idempotent without idempotency key |
| Backoff | Exponential with jitter |
| Max Attempts | 3-5 typical |
Example: Exponential Backoff + Jitter
RetryConfig cfg = RetryConfig.custom()
.maxAttempts(4)
.intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
Duration.ofMillis(200), 2.0, 0.5))
.retryOnException(e -> e instanceof IOException || e instanceof TimeoutException)
.build();
Warning: Naive retries cause retry storms during outages. Always use jitter and cap retries — and disable retries for upstream errors that came from a downstream retry.
4. Retry Strategy Configuration
| Strategy | Formula | Use |
|---|---|---|
| Fixed | delay = c | Simple polling |
| Linear | delay = n × c | Mild backoff |
| Exponential | delay = c × 2^n | Most common |
| Exp + Jitter | delay = rand(0, c × 2^n) | Best — avoids thundering herd |
| Decorrelated Jitter | delay = rand(c, prev × 3) | AWS-recommended |
5. Timeout Pattern
| Layer | Typical Value |
|---|---|
| Connect Timeout | 1-2s |
| Read/Request Timeout | 500ms - 5s (per-call) |
| Total Deadline | Set at edge; propagated downstream |
| DB Query | Statement timeout (e.g., Postgres statement_timeout) |
6. Fallback Pattern
| Strategy | Example |
|---|---|
| Cached | Return last-good cached value |
| Default | Empty array, neutral value |
| Stub | Hardcoded safe response |
| Alternative Source | Secondary service / read replica |
| Graceful Degradation | Hide feature in UI |
7. Fail Fast Pattern
| Aspect | Detail |
|---|---|
| Principle | Reject immediately rather than queuing/blocking |
| Triggers | Open circuit, full bulkhead, exceeded rate limit |
| Response | HTTP 503 with Retry-After |
| Benefit | Frees resources; signals client to back off |
8. Graceful Degradation Pattern
| Layer | Strategy |
|---|---|
| UI | Hide non-essential widgets if backend down |
| API | Return partial response with errors in payload |
| Service | Use cache instead of live data |
| DB | Read-only mode if writes failing |
9. Compensating Transaction Pattern
| Aspect | Detail |
|---|---|
| Use Within Resilience | Undo partial work after non-recoverable failure |
| Differs From Rollback | Logical undo (refund) not transactional |
| Trigger | Saga failure, downstream irreversible failure |
10. Shed Load Pattern
| Mechanism | Detail |
|---|---|
| Detect Overload | Queue depth, latency p99, CPU/mem |
| Reject | Drop low-priority requests with 503 |
| Prioritize | Premium / health checks pass through |
| Adaptive | Use TCP-Vegas-like control to find load ceiling |