Implementing Distributed Coordination
1. Understanding Coordination Services
| Service | Backend | API Style |
|---|---|---|
| ZooKeeper | Zab | Hierarchical znodes |
| etcd | Raft | Key-value + watch |
| Consul | Raft | KV + service catalog |
| Chubby | Paxos | Internal Google lock service |
2. Implementing Barrier Synchronization
| Step | Action |
|---|---|
| 1 | Each participant creates ephemeral child under /barrier |
| 2 | When child count == N, all may proceed |
| 3 | Use watches to be notified of count change |
3. Implementing Distributed Locks
Example: Redis SET NX lock with token
String token = UUID.randomUUID().toString();
String resp = jedis.set("lock:order:42", token,
SetParams.setParams().nx().px(10_000));
if ("OK".equals(resp)) {
try { criticalSection(); }
finally {
// Release only if we still own it
String lua = "if redis.call('get', KEYS[1]) == ARGV[1] then "
+ "return redis.call('del', KEYS[1]) else return 0 end";
jedis.eval(lua, List.of("lock:order:42"), List.of(token));
}
}| Implementation | Notes |
|---|---|
| Redis SET NX PX | Single-instance; needs Redlock or fencing for safety |
| Redlock | Multiple Redis instances; debated correctness |
| ZooKeeper recipe | Ephemeral sequential znode |
| etcd lease + lock | Strong, with revision as fencing token |
Warning: Distributed locks are not safe without fencing tokens — GC pauses or network delays can cause two clients to think they hold the lock simultaneously.
4. Implementing Distributed Semaphores
| Approach | Detail |
|---|---|
| N child znodes | Acquire = create ephemeral; release = delete |
| etcd transactions | CAS on counter with bound |
| Redis | Lua script with INCR + CAS |
5. Implementing Distributed Counters
| Strategy | Trade-off |
|---|---|
| Single-key INCR | Strong; bottleneck at key |
| Sharded counters | Sum N keys on read; scales writes |
| G-Counter (CRDT) | Mergeable; eventually consistent |
| HyperLogLog | Approximate distinct count |
6. Implementing Group Membership Management
| Mechanism | Detail |
|---|---|
| Ephemeral nodes | ZooKeeper auto-removes on disconnect |
| Lease + heartbeat | etcd / Consul |
| Gossip membership | SWIM (Serf, Memberlist) |
7. Implementing Distributed Queues
| Backend | Pattern |
|---|---|
| ZooKeeper | Sequential znodes; consumers watch lowest |
| Redis Streams | Consumer groups, XREADGROUP, ACK |
| Kafka | Partitioned log + consumer groups |
| SQS / Pub/Sub | Managed at-least-once |
8. Implementing Configuration Management
| Pattern | Detail |
|---|---|
| Watch-based | Clients subscribe to key changes (etcd watch) |
| Versioned KV | Atomic updates, rollback by revision |
| Push vs pull | Push reduces latency; pull simpler |
| Tools | Consul KV, Spring Cloud Config, etcd |
9. Implementing Distributed Barriers
| Variant | Use |
|---|---|
| Entry barrier | Wait for N participants to arrive |
| Exit barrier | Wait for all to finish phase before next |
| Double barrier | Both entry and exit |
| Use case | MapReduce phases, batch jobs |
10. Understanding Coordination Trade-offs
| Trade | Detail |
|---|---|
| Strong vs eventual | Coordination = serialization point |
| Centralized service | Simplifies semantics; SPOF if not HA |
| Coordination cost | Round-trip per op; minimize via batching |
| Avoid when possible | Idempotency & CRDTs reduce need |