Week 4: Distributed Log Storage
When Log Storage Stops Being a Database Problem
Every production observability platform eventually discovers that storing logs is not a CRUD exercise—it is a distributed systems problem. Writes must survive node failures. Partitions must exist before traffic arrives. Coordinators must know who leads. Queries must fan out without blocking. And replicas that drift must repair themselves before operators lose trust in the data.
What We Built
A single Spring Boot application hosting a complete distributed log storage stack: replication gateways, partitioned ingestion, consistent-hash routing, Raft leadership, gossip membership, scatter-gather queries, tunable quorums, and anti-entropy repair
RocksDB-backed local storage with quorum replication (W=2 of 3 replicas) and Resilience4j circuit breakers on gateway paths
PostgreSQL composite partitioning (daily ranges × 256 source-hash sub-partitions) with optional Kafka ingestion and batch writes
An operations dashboard at
/aggregating Micrometer metrics, cluster membership, and a one-click demo that exercises every subsystem
Why This Matters
At Netflix or Datadog scale, log platforms ingest billions of events per day. No single node holds the data. No single query scans every shard. The engineering challenge is composing replication, partitioning, consensus, and repair into something operators can reason about under failure. This project implements that composition in one JVM—sharing H2/PostgreSQL persistence, optional Redis topology, and Prometheus export—so you can see how production patterns interact rather than studying them in isolation.
System Design Deep Dive
Multi-node replication with quorum writes. The storage cluster persists LogEntry records in RocksDB and propagates ReplicationRequest payloads to peers via RestTemplate. A write succeeds only when storage.replication.write-quorum acknowledgments arrive within the configured timeout. The trade-off is explicit: lower quorum improves availability during partitions; higher quorum improves durability.
Consistent hashing for load balance. ConsistentHashRing places 150 virtual nodes per physical node using MurmurHash3, minimizing key movement when nodes join or leave—unlike naive modulo sharding that reshuffles everything. DistributionMetricsService exposes storage.distribution.balance_score so operators catch hot spots before they become incidents.
Raft for ordered leadership. RaftNode runs election timeouts (150–300 ms) and leader heartbeats on a dedicated ScheduledExecutorService—sub-second timing that Spring’s @Scheduled pool is not tuned for. Only the leader accepts writes; followers redirect until leadership stabilizes after failover.
Tunable consistency via quorums. QuorumService honors ConsistencyLevel (ONE, QUORUM, ALL) with version-vector conflict resolution. ONE minimizes latency; ALL maximizes durability at the cost of write availability during outages.
Distributed Log Storage Platform Architecture
