High Availability Is a Pipeline, Not a Checkbox
When a log platform loses its primary writer, the failure is rarely “the database went down.” It is a cascade: heartbeats expire, lag spikes, retries amplify load, and the circuit that should protect Postgres opens too late—or never. Production teams at Netflix, Uber, and Datadog treat availability as layered controls on the ingest path, not a single failover script. This Spring Boot system encodes that mindset in one JVM: every event walks through backpressure, leadership, multi-region dedup, circuit-guarded persistence, and optional Kafka publish before it is accepted.
What We Built
A single Spring Boot 3.2 application (
platform-ha-ft) with five HA/FT packages—failover, replication, circuit breaker, backpressure, and chaos—composed into one unified ingest pipelineProfile-driven infrastructure (
local,kafka,docker,redis) so the same JAR runs on H2/in-memory stores or against ZooKeeper, Kafka, Redis, Postgres, and PrometheusActive-passive leadership with epoch fencing, Bloom + exact deduplication, watermark reorder buffers, Resilience4j breakers with a bounded write buffer, and lag-adaptive rate limiting
A chaos orchestrator that validates steady state, injects simulated failures, and proves recovery—plus Micrometer metrics scraped by Prometheus
Why This Matters
At FAANG-scale observability platforms, ingest never trusts a single hop. A region outage must not duplicate events forever; a Postgres blip must not drop the write path; a traffic spike must reject early rather than thrash every downstream dependency. The anti-pattern is bolting “failover” onto an otherwise happy-path pipeline and hoping demos survive production. This project forces those controls into the request path so you can see latency, rejection, buffering, and recovery as first-class outcomes—not afterthoughts.
System Design Deep Dive
Active-passive failover with epoch fencing. FailoverCoordinator uses Curator LeaderLatch when ZooKeeper is enabled (/platform/failover/leader), otherwise assumes in-memory leadership. On promotion it bumps epoch, recovers FailoverConsumerState, waits briefly, then sets processingAllowed=true. Standby instances accept ingest as buffered without local processing—avoiding split-brain writes. Heartbeats land in the state store with TTL; demotion flips the gauge platform.failover.leader.status.
Multi-region replication without double-counting. DeduplicationService combines a Guava Bloom filter with exact setIfAbsent keys keyed by region and event ID. ReorderBufferService holds events for a configurable watermark window (app.replication.reorder.window-ms) so late arrivals persist with a distinct processing layer. RegionReplicationService mirrors to the peer region on a schedule—simulating cross-region MirrorMaker-style catch-up inside one process.
Circuit breakers that degrade, not die. ResilientPersistenceService wraps JPA saves with postgresWriter and postgresRetry. When the breaker opens, writes spill into a 500-event buffer; WriteBufferReconciler drains it every 20 seconds. Redis cache access uses a separate redisCache breaker. Availability demos force OPEN states to prove ingest still returns accepted/buffered instead of 500s.
Backpressure before amplification. BackpressureGate sits first in PlatformIntegrationPipelineService: Resilience4j rate limiter at 1000 permits/second with a 100ms timeout. LagMonitoringService adapts limits from queue depth proxies—healthy / degraded / critical—so spikes produce HTTP 429 rather than unbounded memory growth.
