What We’re Building Today
A real-time ML inference pipeline that classifies incoming logs by severity (DEBUG/INFO/WARN/ERROR/FATAL) and category (Security, Performance, Business, Infrastructure)
A feature extraction service that transforms raw log strings into numeric vectors using TF-IDF and regex-driven heuristics
A model serving layer backed by a Spring Boot microservice with Redis-cached predictions and Kafka-driven async classification
A feedback loop that persists misclassification signals to PostgreSQL for future model retraining
Why This Matters
Production log volumes at scale are unmanageable without automated triage. Netflix processes ~2 billion log events per day. At that volume, manually routing logs to the right team—security incidents to SecOps, latency spikes to SRE, billing anomalies to Finance—is impossible. Classification latency directly impacts Mean Time to Detect (MTTD): a model that takes 500ms to classify a FATAL event is operationally useless during an outage.
The deeper challenge isn’t accuracy—it’s inference throughput with bounded latency. You need a model that classifies 50,000 logs/second while staying under 10ms p99. This forces architectural choices: batch vs. stream inference, model size vs. speed trade-offs, and whether to embed the model in the service or call an external model server.
Today’s implementation reveals why Uber’s ML Platform (Michelangelo) separates feature stores, model serving, and training pipelines as independent concerns—and why you should too.
System Design Deep Dive
1. The Feature Engineering Bottleneck
Raw log strings are not model inputs. Converting them is your first scalability problem.
Naive approach: Run TF-IDF over the entire log corpus at inference time. This is O(n × vocab_size) and breaks under load.
Production approach: Pre-compute a fixed vocabulary offline. At inference time, apply the vocabulary as a lookup table—O(vocab_size) with a cap. Combine with hand-crafted features: HTTP status code presence, stack trace depth, keyword density for known error patterns (NullPointerException, OutOfMemoryError, ECONNREFUSED). These heuristics are cheap, interpretable, and surprisingly powerful.
Architectural insight: Features are data, not code. Store your vocabulary and normalization stats in Redis, not in the JVM heap. This means zero-downtime vocabulary updates and shared state across multiple classifier instances. When you eventually move to embeddings (sentence-transformers, etc.), the same pattern applies—the feature store is your contract between training and serving.
Trade-off: Heuristic features are brittle. They over-fit to your log format. When a downstream service changes its log format, your classifier silently degrades. You need drift detection—monitoring the distribution of input features over time and alerting when it shifts beyond a threshold (KL-divergence or Population Stability Index).
