What We’re Building Today
Near-real-time indexing pipeline that indexes logs within 100ms of arrival
Distributed inverted index with LSM-tree optimization for high write throughput
Index coordination layer managing shard distribution and replication across nodes
Query API with millisecond-latency searches over freshly indexed data
Why This Matters
Search latency directly impacts incident response time. When a production outage occurs, every second engineers spend waiting for logs to become searchable costs money and customer trust. Netflix processes 1 trillion events daily—if their indexing pipeline had even a 10-second delay, teams would be flying blind during critical incidents. Real-time indexing transforms logs from historical artifacts into actionable intelligence.
The challenge isn’t just speed—it’s maintaining search quality while ingesting 50,000+ events per second. Traditional batch indexing offers perfect consistency but unacceptable latency. Stream-based indexing delivers speed but introduces complexity around partial updates, segment management, and query consistency. Today’s architecture balances these trade-offs using proven patterns from Elasticsearch, Splunk, and Datadog.
System Design Deep Dive
