Understanding Distributed Systems Concepts
1. Understanding Distributed Computing Challenges
| Challenge | Description | Mitigation |
|---|---|---|
| Partial failure | Some nodes fail, others continue | Timeouts, retries, idempotency |
| Network unreliability | Packets lost/delayed/reordered | TCP, retries, ack protocols |
| Concurrency | Multiple nodes act simultaneously | Locks, consensus, CRDTs |
| No global clock | Cannot order events globally | Logical clocks, NTP+TrueTime |
| Heterogeneity | Mixed HW/OS/lang versions | Standard protocols, schemas |
| Security | Untrusted network | mTLS, signed messages |
2. Understanding the Fallacies of Distributed Computing
| # | Fallacy | Reality |
|---|---|---|
| 1 | Network is reliable | Packets lost, partitions occur |
| 2 | Latency is zero | Light-speed lower bound; cross-region 100ms+ |
| 3 | Bandwidth is infinite | Bottleneck under load |
| 4 | Network is secure | Always assume hostile |
| 5 | Topology doesn't change | Auto-scaling, deploys, failures |
| 6 | One administrator | Multi-team, multi-cloud |
| 7 | Transport cost is zero | Egress costs $$, serialization CPU |
| 8 | Network is homogeneous | Mixed protocols, devices, versions |
3. Understanding Network Partitions and Failures
| Failure Type | Symptom | Detection |
|---|---|---|
| Crash failure | Node stops responding | Heartbeat timeout |
| Omission failure | Some messages dropped | Acks + retries |
| Network partition | Subset cannot reach others | Quorum loss detection |
| Byzantine failure | Arbitrary/malicious behavior | BFT consensus |
| Gray failure | Slow but not failed | p99 latency, hedged requests |
4. Understanding Clock Synchronization
| Clock | Mechanism | Accuracy |
|---|---|---|
| NTP | Public time servers | ~10–100ms |
| PTP | HW timestamps in DC | <1µs |
| GPS/Atomic | Direct hw reference | ns range |
| TrueTime (Spanner) | GPS+atomic+uncertainty bound | <7ms uncertainty |
Warning: Never rely on wall clocks for ordering across nodes — clock skew, drift, and leap seconds make it unsafe.
5. Understanding Vector Clocks and Lamport Timestamps
| Type | Captures | Limitation |
|---|---|---|
| Lamport timestamp | Single counter, total order | Cannot detect concurrent events |
| Vector clock | Per-node counters; partial order | O(N) size, GC complex |
| Version vector | Replica versioning for conflicts | Used in Dynamo, Riak |
| Hybrid Logical Clock | Physical + logical bits | Bounded skew, used in CockroachDB |
Example: Vector clock causality
Node A: [A:1] → send msg → Node B receives
Node B: merges = [A:1, B:1] → send to C
Node C: [A:1, B:1, C:1]
Concurrent: [A:2, B:0] vs [A:0, B:2] (neither dominates)6. Understanding Distributed Transaction Challenges
| Challenge | Cause | Solution |
|---|---|---|
| Atomicity across nodes | Partial commits | 2PC, 3PC, Saga |
| Coordinator failure | Blocks participants | Timeouts, 3PC, Paxos commit |
| Performance | Many round trips | Batch, optimistic concurrency |
| Cross-DB transactions | XA limited support | Sagas, outbox pattern |
7. Understanding Two-Phase Commit (2PC)
2PC Protocol
- Prepare: Coordinator sends PREPARE; participants vote YES (write to log) or NO
- Commit: If all YES → COMMIT; any NO → ABORT; participants apply and ACK
| Property | Behavior |
|---|---|
| Atomicity | All commit or all abort |
| Blocking | If coordinator dies after PREPARE, participants block |
| Latency | 2 round trips + log fsyncs |
| Used in | XA transactions, distributed RDBMS |
8. Understanding Three-Phase Commit (3PC)
| Phase | Action |
|---|---|
| CanCommit | Coordinator asks participants if they can commit |
| PreCommit | If all yes, coordinator says "prepare to commit" |
| DoCommit | Final commit; participants ACK |
Note: 3PC is non-blocking with synchronous network but unsafe under network partitions. Modern systems prefer Paxos/Raft commit.
9. Understanding Distributed Snapshots
| Algorithm | Purpose | Use Case |
|---|---|---|
| Chandy-Lamport | Consistent global snapshot via markers | Deadlock detection, checkpointing |
| Asynchronous Barrier Snapshot | Snapshot in stream pipeline | Apache Flink checkpointing |
| Copy-on-write | Point-in-time view of state | ZFS, etcd snapshots |
10. Understanding Byzantine Fault Tolerance
| Algorithm | Tolerates | Use Case |
|---|---|---|
| PBFT | (N-1)/3 malicious nodes | Hyperledger, permissioned chains |
| Tendermint | 1/3 byzantine | Cosmos SDK |
| PoW (Nakamoto) | <50% hash power | Bitcoin |
| PoS variants | <1/3 stake byzantine | Ethereum, Solana |
Note: BFT requires 3f+1 nodes to tolerate f byzantine failures. Used where actors may be malicious (blockchains, multi-org systems). Not needed in trusted DCs.