Implementing Data Partitioning
1. Understanding Partitioning Strategies
| Strategy | Mechanism | Trade-offs |
|---|---|---|
| Hash | partition = hash(key) mod N | Even distribution; bad for range queries |
| Range | Contiguous key ranges per partition | Range scans efficient; hot ranges possible |
| Consistent Hash | Hash ring with virtual nodes | Minimal data movement on rebalance |
| Directory / Lookup | External metadata maps key → node | Flexible; lookup overhead, SPOF |
| Composite | Hash(tenant) + range(time) | Combines benefits |
2. Implementing Consistent Hashing
Example: Consistent hash ring with virtual nodes
public class ConsistentHashRing<T> {
private final SortedMap<Long, T> ring = new TreeMap<>();
private final int vnodesPerNode;
public ConsistentHashRing(int vnodesPerNode) { this.vnodesPerNode = vnodesPerNode; }
public void addNode(T node) {
for (int i = 0; i < vnodesPerNode; i++) {
ring.put(hash(node + "#" + i), node);
}
}
public T getNode(String key) {
if (ring.isEmpty()) return null;
long h = hash(key);
SortedMap<Long, T> tail = ring.tailMap(h);
Long target = tail.isEmpty() ? ring.firstKey() : tail.firstKey();
return ring.get(target);
}
private long hash(String s) {
return Hashing.murmur3_128().hashString(s, StandardCharsets.UTF_8).asLong();
}
}
| Property | Value |
|---|---|
| Data movement on add/remove | K/N keys (vs K with mod N) |
| Recommended vnodes | 100-256 per physical node |
| Hash function | MurmurHash3, xxHash, SHA-1 |
3. Implementing Virtual Nodes (vnodes)
| Benefit | Detail |
|---|---|
| Load balance | Smooths skew across physical nodes |
| Heterogeneous capacity | Larger nodes get more vnodes |
| Faster rebalance | Many small transfers, parallel |
| Used by | Cassandra, DynamoDB, Riak |
4. Implementing Hash Slots (Redis Cluster)
| Property | Value |
|---|---|
| Total slots | 16384 (2¹⁴) |
| Slot computation | CRC16(key) mod 16384 |
| Hash tags | {user:42}.profile — only braces hashed for co-location |
| Resharding | Slot migration via MIGRATE |
5. Implementing Range Partitioning
| Aspect | Detail |
|---|---|
| Layout | Sorted key ranges per partition (e.g., [a-f], [g-m]) |
| Strength | Range scans, ordered iteration |
| Risk | Hot range (sequential keys → one shard) |
| Auto-split | Splitter divides hot range into two |
| Used by | HBase, BigTable, Spanner, CockroachDB |
6. Handling Partition Rebalancing
| Strategy | Mechanism | Pros / Cons |
|---|---|---|
| Hash mod N | Reassign all keys | Simple; massive movement |
| Fixed partitions | 10× nodes; reassign whole partitions | Predictable; coarse |
| Dynamic split | Split when partition exceeds size | Adapts to growth (HBase) |
| Consistent hash | Move only affected vnodes | Minimal movement |
Warning: Avoid automatic rebalancing during peak traffic — schedule or rate-limit transfers.
7. Understanding Hot Spots and Skewed Workloads
| Cause | Mitigation |
|---|---|
| Celebrity user / tenant | Salt key (userId#shardN) and fan-in on read |
| Sequential timestamps | Hash prefix the time key |
| Single trending entity | Cache + replicate hot keys |
| Skewed range | Auto-split hot range |
8. Selecting Partition Keys
| Criterion | Why |
|---|---|
| High cardinality | Spreads load |
| Even access pattern | Avoids hot shards |
| Aligns with query | Reduces cross-partition scatter-gather |
| Stable | Avoids re-sharding |
9. Handling Cross-Partition Queries
| Pattern | Approach |
|---|---|
| Scatter-gather | Fan out to all partitions, merge |
| Parallel aggregation | Map-reduce style |
| Materialized view | Pre-compute by alternate key |
| Global secondary index | Separate partitioning by index key |
10. Implementing Secondary Indexes in Partitioned Systems
| Type | Layout | Trade-offs |
|---|---|---|
| Local (document-partitioned) | Index lives with data partition | Cheap writes; scatter-gather reads |
| Global (term-partitioned) | Index partitioned by index key | Fast reads; distributed writes |
| Examples | DynamoDB GSI vs LSI; ES per-shard inverted index | — |