Understanding Distributed Systems Fundamentals
1. Understanding Distributed System Definition
| Property | Definition | Example |
|---|---|---|
| Distributed System | Collection of independent nodes appearing as single coherent system | Kubernetes cluster, Cassandra, Kafka |
| Node | Independent computing entity (process, VM, container) | Pod, EC2 instance, broker |
| Cluster | Group of nodes coordinating to provide service | 3-node etcd cluster |
| Coordination | Mechanisms for nodes to agree on shared state | Consensus, locks, leader election |
| Shared-Nothing | Each node has independent CPU/memory/disk | Cassandra, BigTable |
2. Understanding Network Communication
| Aspect | Detail | Impact |
|---|---|---|
| Message Passing | Only way nodes interact | No shared memory across nodes |
| Latency | Time for round-trip; ~0.5ms LAN, 50-200ms WAN | Bounds responsiveness |
| Bandwidth | Throughput limit per link | Constrains data volume |
| Packet Loss | 0.01-1% typical; higher under congestion | Requires retries/ACKs |
| Ordering | TCP per-connection only; UDP none | App must enforce global order |
| Protocols | TCP, UDP, QUIC, HTTP/2, HTTP/3, gRPC | Choose by latency/reliability |
3. Understanding Partial Failures
| Failure Type | Description | Detection |
|---|---|---|
| Crash-Stop | Node halts permanently | Heartbeat timeout |
| Crash-Recovery | Node halts, then resumes (possibly stale) | Epoch/term numbers |
| Omission | Messages lost in transit | Sequence numbers, ACKs |
| Network Partition | Subset of nodes unreachable | Quorum loss, gossip divergence |
| Byzantine | Arbitrary/malicious behavior | Cryptographic verification |
| Gray Failure | Slow but not down (degraded) | Latency percentiles, anomaly detection |
Warning: Cannot distinguish slow node from failed node from a remote vantage point — only timeouts approximate this.
4. Understanding Transparency Goals
| Type | Hides From User | Example |
|---|---|---|
| Access | Local vs remote resource access | NFS, gRPC stubs |
| Location | Physical placement | DNS, service discovery |
| Migration | Resource moving between nodes | Pod rescheduling |
| Replication | Multiple copies exist | S3, DynamoDB |
| Concurrency | Other users sharing resource | DB transactions |
| Failure | Component failures recovered transparently | RAID, retries, failover |
| Scaling | System growth/shrinkage | Auto-scaling groups |
5. Understanding Fallacies of Distributed Computing
| # | Fallacy | Reality |
|---|---|---|
| 1 | Network is reliable | Packets drop, links fail |
| 2 | Latency is zero | Speed-of-light bounds; cross-continent ~150ms |
| 3 | Bandwidth is infinite | Costs and physical limits |
| 4 | Network is secure | Assume hostile; use TLS, mTLS |
| 5 | Topology doesn't change | Nodes join/leave constantly |
| 6 | One administrator | Multi-team, multi-cloud |
| 7 | Transport cost is zero | Egress/serialization expensive |
| 8 | Network is homogeneous | Mixed protocols, MTUs, vendors |
6. Understanding Scalability Types
| Type | Approach | Limit |
|---|---|---|
| Vertical (scale up) | Bigger CPU/RAM/disk on one node | Hardware ceiling, cost ∝ size² |
| Horizontal (scale out) | Add more nodes | Coordination overhead |
| Functional | Split by service/feature | Service boundaries |
| Data (sharding) | Partition data across nodes | Cross-shard queries |
| Geographic | Distribute across regions | Cross-region consistency |
7. Understanding Latency vs Throughput Tradeoffs
| Metric | Definition | Optimization |
|---|---|---|
| Latency | Time for single request (p50/p95/p99) | Caching, fewer hops, faster paths |
| Throughput | Requests/second handled | Batching, parallelism, async |
| Tail Latency | p99/p999 — outliers | Hedged requests, tied requests |
| Little's Law | L = λW (concurrency = rate × latency) | Bound queue depth |
Note: Optimizing throughput (batching) often hurts latency; pick targets per workload.
8. Understanding Reliability vs Availability
| Metric | Formula | Target |
|---|---|---|
| Availability | Uptime / (Uptime + Downtime) | 99.9% = 8.76h/yr down |
| 99.99% (4 nines) | — | 52.6 min/yr |
| 99.999% (5 nines) | — | 5.26 min/yr |
| MTBF | Mean Time Between Failures | Higher = more reliable |
| MTTR | Mean Time To Recover | Lower = better availability |
| Reliability | P(no failure in time T) | Function of MTBF |
9. Understanding Distributed System Challenges
| Challenge | Cause | Mitigation |
|---|---|---|
| Partial Failure | Some nodes down, others up | Replication, retries, circuit breakers |
| Network Partitions | Split connectivity | Quorum, partition-tolerant protocols |
| Clock Skew | No global time | Logical clocks, HLC, NTP |
| Concurrency | Simultaneous updates | Locks, MVCC, CRDTs |
| Ordering | Messages arrive out-of-order | Sequence numbers, vector clocks |
| Consensus | Agreement under failure | Paxos, Raft |
| Debuggability | State spread across nodes | Tracing, structured logs |
10. Understanding When to Use Distributed Systems
| Use When | Avoid When |
|---|---|
| Single node cannot meet load/storage | Single node suffices |
| HA > 99.9% required | Best-effort SLA acceptable |
| Geographic locality needed | Single region OK |
| Independent team/service deploys | Monolith team |
| Fault isolation between components | Coupled lifecycles fine |
Warning: Distributed systems multiply operational complexity 10×. Start with a monolith; distribute when forced by scale, availability, or org boundaries.