Week 11: Performance Optimization — Make every log byte earn its keep
What we’re building today
When Cloudflare ingests 10M+ HTTP logs per second, they don’t write one event at a time—they batch, compress, and skip duplicates before touching disk. Datadog caches hot queries in memory tiers. ClickHouse picks columnar storage because analytics scans columns, not rows. This lesson wires seven performance layers into one platform.
By the end you’ll have:
Pipeline profiler with bottleneck detection
Self-tuning adaptive batching
L1 LRU query cache
Row / columnar / hybrid storage selection
Bloom filter deduplication
Delta encoding with zlib compression
Adaptive worker scaling
Why this matters
Kafka producers batch messages because each network round-trip costs milliseconds—your adaptive batcher learns the sweet spot automatically.
Redis sits in front of Postgres for hot keys—your L1 cache does the same for repeated log queries.
S3 lifecycle rules move cold data to cheaper tiers—your storage optimizer picks row vs columnar based on access patterns.
Cassandra uses bloom filters to skip SSTable reads—your bloom layer skips duplicate log fingerprints at ingest.
InfluxDB delta-encodes timestamps—your encoder stores only what changed since the last baseline.
Performance is not one trick—it is a stack of small decisions that compound.
Core concepts
Architecture
Single FastAPI service on port 8095. All performance modules run in-process with SQLite persistence.
Preparing for a distributed systems interview?
→Download the free Interview Pack
→ Subscribe now to access source code repository - 200 + coding lessons
