Implementing Load Balancing
1. Configuring Round Robin Algorithm
| Property | Value |
|---|---|
| Distribution | Even, rotational |
| State | Stateless counter |
| Best for | Homogeneous backends |
| Limitation | Ignores load/latency |
Example: Round-robin upstream
upstream backend {
server svc-1.internal:8080;
server svc-2.internal:8080;
server svc-3.internal:8080;
}2. Using Least Connections Algorithm
| Property | Value |
|---|---|
| Distribution | Pick lowest active conns |
| State | Per-target counter |
| Best for | Long-lived requests |
| NGINX directive | least_conn; |
3. Using IP Hash Algorithm
Example: IP hash for session affinity
upstream backend {
ip_hash;
server svc-1.internal:8080;
server svc-2.internal:8080;
}| Property | Value |
|---|---|
| Hash key | Client IP |
| Stickiness | Same IP → same backend |
| Gotcha | NAT, mobile users skew |
4. Implementing Weighted Load Balancing
| Target | Weight | Share |
|---|---|---|
| large.svc | 4 | 50% |
| medium.svc | 2 | 25% |
| small.svc | 2 | 25% |
5. Configuring Consistent Hashing
| Property | Value |
|---|---|
| Algorithm | Ring hash (Ketama/Maglev) |
| Hash input | Header, cookie, query, path |
| Re-balance cost | O(1/N) keys move on change |
| Best for | Cache locality, sharding |
Example: Consistent hash by user ID (Envoy)
load_balancing_policy:
policies:
- typed_extension_config:
name: envoy.load_balancing_policies.ring_hash
typed_config:
"@type": type.googleapis.com/envoy.extensions.load_balancing_policies.ring_hash.v3.RingHash
minimum_ring_size: 1024
hash_policy:
- header: { header_name: x-user-id }6. Implementing Sticky Sessions
| Method | Mechanism |
|---|---|
| Cookie insert | GW adds SERVERID cookie |
| Cookie prefix | Modify existing app cookie |
| App cookie | Hash app-set cookie |
| Header | Hash custom header |
Warning: Sticky sessions break horizontal scaling and graceful drain. Prefer stateless sessions (JWT, Redis).
7. Setting Load Balancing Weights
| Scenario | Weight Setup |
|---|---|
| Heterogeneous instances | 2x CPU → 2x weight |
| Canary | Stable=99, canary=1 |
| Drain | Target weight → 0 |
| Cost balancing | Spot=70, on-demand=30 |
8. Setting Up Active Health Checks
Example: Envoy active health check
health_checks:
- timeout: 1s
interval: 5s
unhealthy_threshold: 3
healthy_threshold: 2
http_health_check:
path: /healthz
expected_statuses: [{ start: 200, end: 299 }]9. Configuring Passive Health Checks
| Trigger | Action |
|---|---|
| 5 consecutive 5xx | Eject 30s |
| Connection reset | Eject 10s |
| Timeout | Eject 30s |
| Local origin failure | Eject + circuit break |
10. Configuring Failover Behavior
Primary OK → Route to primary
Primary FAIL → Retry on primary (n=2)
Still FAIL → Switch to secondary pool
Secondary OK → Stay until primary healthy 30s
| Strategy | Use |
|---|---|
| Active-active | Both serve traffic |
| Active-passive | Standby only on failure |
| Priority groups | Tier 1 → tier 2 fallback |