What We’re Building Today
RetentionPolicyEngine — a configurable, rule-based system that maps log categories to retention windows
TieredStorageManager — automatic promotion/demotion of log data across hot → warm → cold storage tiers
RetentionEnforcerJob — a scheduled, idempotent enforcement job that purges expired partitions safely
PolicyMetricsCollector — observability layer reporting retention lag, purge throughput, and storage reclamation rates
Why This Matters
Every byte you store costs money. Every byte you keep longer than legally required is a liability. Data retention is where economics, compliance, and system design collide.
At Uber’s scale, the ride-events topic alone generates terabytes per day. Without automated retention enforcement, storage costs compound weekly. Netflix’s content access logs must be purged within defined windows to satisfy GDPR and CCPA — a manual process would require a dedicated team and still be error-prone.
The naive approach — a cron job that DELETEs rows matching WHERE created_at < NOW() - INTERVAL '90 days' — will destroy your database under production load. A full table scan locks rows, spikes I/O, and triggers autovacuum storms in PostgreSQL. The production pattern is fundamentally different: you don’t delete data, you expire partitions.
Tomorrow we layer GDPR “right to be forgotten” on top of today’s time-based retention. Today’s architecture is deliberately designed to make that addition surgical rather than invasive.
System Design Deep Dive
Preparing for a distributed systems interview?
→Download the free Interview Pack
→ Subscribe now to access source code repository - 200 + coding lessons
