Week 8: Distributed Log Search — “Find the needle in the haystack”
What we’re building today
When Cloudflare engineers hunt a bad deploy, they don’t
grepterabytes—they query an inverted index built in milliseconds. Splunk, Elasticsearch, and Loki all share the same spine: tokenize → shard → index → rank → expose an API. This week you wire seven search layers into one platform.By the end you’ll have:
Inverted index with posting lists
Four-shard consistent-hash distribution
SQL-like query parser and executor
Faceted filters on service, level, region
Sub-second realtime segment indexing
TF-IDF + recency + severity ranking
REST search API + React dashboard
Why this matters
Elasticsearch inverted indices power Kibana—your posting lists are the same data structure at smaller scale.
Splunk maps search terms to index buckets via consistent hashing—identical to Day 53’s ring.
Grafana Loki streams labels into facets—your facet engine mirrors that label model.
Datadog Log Management ranks ERROR logs above DEBUG during incidents—your ranking weights do the same.
Search is how operators turn logs into answers under pressure.
Core concepts
Architecture
Single FastAPI service on port 8092. SQLite stores logs and facet metadata. Shards are in-process partitions—same coordination logic as multi-node without extra containers.
Preparing for a distributed systems interview?
→Download the free Interview Pack
→ Subscribe now to access source code repository - 200 + coding lessons
