What We’re Building Today
Partitioned search index spanning 3+ independent index nodes using consistent hashing
Scatter-gather query coordinator that parallelizes searches across shards
Index replication layer ensuring fault tolerance with primary/replica topology
Shard routing service that maps log entries to appropriate index partitions based on tenant ID
Why This Matters
Yesterday’s inverted index hit a wall at ~500GB of log data. Single-node indexes face three critical bottlenecks: index size exceeds available RAM forcing disk swaps, write throughput caps at disk I/O limits (~10K writes/sec), and query latency degrades linearly with index growth. Elasticsearch solves this by distributing indexes across hundreds of nodes, processing 10M+ documents per second. LinkedIn’s distributed search infrastructure handles 15TB of indexed data across 200+ nodes, serving queries in <50ms p99. Today we’re implementing the core pattern that powers these systems: horizontal index partitioning with coordinated scatter-gather queries.
The architectural insight: distributed indexes trade consistency for availability and partition tolerance. When you shard an index, you’re choosing AP over CA in the CAP theorem—accepting eventual consistency in exchange for horizontal scalability and fault tolerance. Understanding this trade-off is fundamental to system design at scale.
System Design Deep Dive
