The difference between a "design interview" and a "production system" is massive. Close that gap today with the Hands on Distributed Log System Building course. Get 40% off for a limited time: https://sdcourse.substack.com/fbbab0d8

What We’re Building Today

• Real-time inverted index that tokenizes and indexes log messages as they arrive via Kafka

• Search API with relevance scoring and ranked results for natural language queries

• Index persistence layer using Redis for hot data and PostgreSQL for cold storage

• Query processing engine supporting boolean operators and phrase matching

Why This Matters

Every major observability platform—Splunk, Datadog, Elastic—runs on inverted indices. When you search “ERROR user authentication failed” across billions of log entries and get results in milliseconds, you’re querying an inverted index. This data structure powers everything from application monitoring to security incident response.

Without inverted indices, log search would require scanning every log entry linearly—O(n) complexity that becomes impossible at scale. An inverted index transforms this into O(k) lookups where k is the number of query terms, enabling sub-second searches across terabytes of logs. Understanding inverted indices is fundamental to building search infrastructure that scales from thousands to trillions of documents.

Today’s implementation bridges the gap between local prototypes and production search engines, showing how the same architectural patterns scale from single-node deployments to distributed clusters processing petabytes daily.

System Design Deep Dive

Read more

• Real-time analytics dashboard consuming aggregated metrics from Kafka streams

• WebSocket-based push architecture delivering sub-second metric updates to browsers

• Multi-dimensional visualization service supporting time-series, histograms, and geographic heatmaps

• Query optimization layer with Redis caching and PostgreSQL time-series partitioning

Why This Matters

At scale, the gap between generating metrics and making them actionable determines your incident response time. Netflix processes 500 billion events daily, but their dashboard systems compress this into 200ms query responses because engineers can’t wait 30 seconds to see if a deployment broke something. When Uber’s surge pricing algorithms trigger, dashboard systems must surface the decision rationale within 100ms or drivers can’t understand why rates changed.

The architectural challenge isn’t building charts—it’s designing systems that maintain query responsiveness as data volume grows exponentially. Your dashboard becomes the bottleneck between detecting problems and fixing them. Poor dashboard architecture means your monitoring system generates alerts 5 minutes before your engineers can see the underlying data.

System Design Deep Dive

Read more

Subscribe to our portal for systemdrd.com & get lifetime access to “Hands On System Design with “Distributed Systems Implementation with python and javascript” and this “Distributed Log Implementation With Java & Spring Boot”

What We’re Building Today

A production-grade distributed alerting system that monitors log patterns in real-time and triggers intelligent notifications:

• Real-time alert rule engine processing 50,000+ events/second with Kafka Streams

• Smart alert manager with deduplication, correlation, and escalation logic

• Multi-channel notification service supporting email, Slack, and PagerDuty integration

• Alert configuration API for dynamic rule management without system restarts

Why This Matters

Alert generation is where distributed log processing transitions from passive observation to active operational response. At scale, naive alerting becomes your biggest operational burden—Netflix processes 2 billion alerts daily but only acts on 0.01% of them. The challenge isn’t detecting problems; it’s preventing alert fatigue while ensuring critical issues never slip through.

Poor alerting architectures create alert storms during outages (compounding incident response), suffer from flapping alerts that erode trust, generate excessive false positives that train teams to ignore notifications, and fail during the very incidents they’re designed to detect. Production alerting requires sophisticated state management, intelligent suppression, and fault-tolerant delivery mechanisms that work when your primary systems are degraded.

System Design Deep Dive

Pattern 1: Stateful Stream Processing for Alert Evaluation

Read more

Today we’re implementing a production-grade anomaly detection system that processes streaming log data to identify unusual patterns in real-time. You’ll build:

• Statistical anomaly detection engine using Z-score and IQR methods for numeric metrics

• Time-series pattern recognition detecting deviations from historical baselines

• Multi-dimensional clustering identifying outliers across correlated log attributes

• Adaptive threshold system that learns normal behavior and adjusts detection sensitivity

• Real-time alerting pipeline with confidence scoring and false-positive suppression

Why This Matters: Production Anomaly Detection at Scale

Anomaly detection is critical infrastructure at companies processing billions of events daily. Netflix’s anomaly detection system monitors 800+ microservices, detecting issues before they impact customer experience. Uber’s real-time fraud detection processes 100,000 trip events per second, identifying suspicious patterns within milliseconds. Amazon’s operational intelligence systems scan millions of metrics to prevent outages.

The challenge isn’t just detecting anomalies—it’s doing so with minimal false positives while maintaining sub-second latency at massive scale. Traditional threshold-based alerting breaks down when you have thousands of metrics with dynamic baselines. Statistical methods provide precision, but require careful tuning for seasonality, trends, and multi-modal distributions.

Today’s implementation demonstrates how to build adaptive anomaly detection that scales horizontally, maintains accuracy under load, and integrates with existing observability infrastructure. The patterns you’ll implement power the monitoring systems behind modern distributed platforms.

System Design Deep Dive

Read more

Today we implement production-grade sessionization to transform raw event streams into meaningful user sessions:

• Session Window Processing: Kafka Streams session windows that automatically group events with configurable inactivity gaps

• Real-Time Session Tracking: Redis-backed active session cache with TTL-based expiration and sub-millisecond lookups

• Session Analytics Engine: PostgreSQL persistence layer computing session metrics (duration, event count, conversion patterns)

• Interactive Query API: REST endpoints exposing session state stores for real-time session queries without external database latency

Why This Matters

Sessionization is the foundation of user behavior analytics at scale. Every time you see “Users who viewed this also bought...” on Amazon, “Continue Watching” on Netflix, or “Complete your ride” on Uber, you’re experiencing sessionization in action. The challenge isn’t just grouping events—it’s doing it correctly with out-of-order events, across millions of concurrent users, while maintaining sub-second query latency.

The distributed systems challenge emerges from time complexity: Events arrive out of order, users cross session boundaries mid-action, and sessions must expire gracefully without memory leaks. Netflix processes 200+ billion events daily across 250 million users, requiring sessionization that handles late-arriving events up to 24 hours delayed while maintaining real-time dashboard updates. Getting this wrong means misattributed user actions, incorrect analytics, and degraded recommendation quality.

System Design Deep Dive

Read more

Stop Drawing Boxes. Start Building Systems. Subscribe Now

The gap between a “system design interview” and a “production system” is massive. This newsletter exists to bridge that gap.

When you join any organisation, no one is going to teach you how system is design or built. Fun part developers rarely document everything. you need to dig through to understand the system. I created this course because I believe the best way to learn distributed systems is by building them. We don’t just talk about the CAP theorem; we look at how it dictates our database choices. We don’t just mention “latency”; we measure it. Subscribe

What We’re Building Today

Today we implement sliding window aggregations for real-time trend detection in distributed log processing systems:

• Hopping windows with configurable slide intervals for continuous metric updates

• Multi-granularity trend analysis tracking 1-minute, 5-minute, and 15-minute moving averages

• State-efficient window management using Kafka Streams’ optimized windowing primitives

• Interactive query API serving real-time trend data with sub-10ms latency

• Production monitoring tracking window lag, state store size, and processing throughput

Why This Matters

Sliding windows solve a critical problem in real-time analytics: detecting trends as they happen. Unlike tumbling windows that update in discrete jumps, sliding windows provide continuous visibility into recent behavior patterns. When Netflix detects video quality degradation, they need second-by-second moving averages—not 5-minute buckets that hide critical spikes. When Uber calculates surge pricing, they track the velocity of ride requests using overlapping windows to smooth out noise while remaining responsive to demand shifts.

The fundamental challenge is maintaining thousands of overlapping windows efficiently. A naive implementation storing every window independently would consume massive memory and CPU. Production systems leverage specialized data structures and time-based compaction strategies to maintain window state efficiently while serving low-latency queries.

System Design Deep Dive

1. Sliding vs Hopping Windows: The Trade-off Space

Sliding windows create a new window for every event, providing maximum granularity but at high computational cost. Hopping windows advance by fixed intervals (the “hop size”), reducing computation while introducing bounded staleness. The key insight: most applications don’t need per-event updates—hopping windows with 10-second hops provide near-continuous trends at 1/100th the cost.

Window Configuration Pattern:

Window Size: 5 minutes
Hop Size: 10 seconds
Result: 30 overlapping windows active simultaneously
Memory: O(window_size / hop_size) per key

Netflix uses 1-minute windows with 5-second hops for video quality metrics, balancing trend detection speed against computational overhead. Each window overlap shares most of its data with adjacent windows, enabling Kafka Streams to optimize through incremental computation rather than reprocessing the full window on every hop.

Anti-Pattern: Setting hop size too small relative to window size. A 1-hour window with 1-second hops creates 3,600 active windows—each requiring state storage and periodic aggregation. The memory footprint becomes O(events_per_second × window_seconds), potentially gigabytes for high-volume streams.

2. State Store Architecture for Window Queries

Kafka Streams materializes windowed aggregations into RocksDB-backed state stores, but querying “what’s the current moving average?” requires understanding how windows are keyed. Each window instance is stored with a composite key: (record_key, window_start_time). To serve a real-time query, we must:

1. Calculate which windows contain the query timestamp

2. Fetch all relevant window instances from the state store

3. Aggregate across windows to compute the moving average

4. Cache the result in Redis for subsequent queries

Critical Design Decision: Window retention time. By default, Kafka Streams retains windows for window_size + grace_period. For a 5-minute window, this means only the last 5-10 minutes are queryable. Longer retention enables historical trend queries but increases state store size linearly.

Uber’s surge pricing system maintains 15 minutes of windowed state to detect both immediate spikes and sustained demand increases. They use a two-tier approach: hot state in RocksDB for recent windows, cold state in S3 for historical analysis.

3. Out-of-Order Event Handling

Real-world data streams are never perfectly ordered. Network delays, producer failures, and buffering create timestamp skew. Kafka Streams handles this through grace periods—extended windows that accept late arrivals for a configured duration after the window would normally close.

The Grace Period Trade-off:

• Too short: Late events dropped, inaccurate trends

• Too long: Increased memory, delayed window finalization

• Production tuning: Set grace period to 95th percentile of observed latency

Twitter’s trending topics system uses a 30-second grace period for their 5-minute trending windows. They found that 30 seconds captures 99% of events while preventing unbounded state growth from severely delayed data. Events arriving later than the grace period are logged to a dead-letter topic for analysis but don’t affect real-time trends.

State Management: Each open window consumes memory proportional to the aggregation size (typically bytes to kilobytes per window). With 10,000 unique keys and 30 windows per key, you’re managing ~300K active window instances. Kafka Streams uses sparse windowing—only creating window instances when events arrive for that key-window pair.

4. Incremental Aggregation Patterns

Computing moving averages requires maintaining both sum and count for each window. The naive approach stores all events in the window and recalculates on every query. Production systems use incremental aggregation:

// Efficient: O(1) per event
windowedStream
  .aggregate(
    () -> new WindowStats(0L, 0L), // sum, count
    (key, value, aggregate) -> {
      aggregate.sum += value;
      aggregate.count++;
      return aggregate;
    }
  );

// Query time: O(1)
double movingAverage = stats.sum / stats.count;

Amazon’s CloudWatch uses this pattern for metric aggregations, maintaining running sums, counts, min, max, and sum-of-squares for percentile calculations. Each metric point is processed exactly once into the window state, then queries read pre-aggregated values.

Failure Handling: Kafka Streams checkpoints window state to changelog topics. On failure, the stream processor restores state from the changelog and resumes processing. Window state is strongly consistent—each window instance exists on exactly one partition, eliminating the need for distributed coordination during aggregation.

5. Query Patterns for Real-Time Dashboards

Serving windowed aggregations requires an interactive query layer. Kafka Streams exposes state stores through ReadOnlyWindowStore interfaces, but querying is local to each stream processor instance. In a multi-instance deployment, you need service discovery to route queries to the correct instance holding the relevant partition.

Production Pattern:

Query Router → [Discovers key partition] → Stream Processor Instance → RocksDB → Response

For globally aggregated metrics (e.g., “average error rate across all services”), you need a scatter-gather approach: query all stream processor instances, aggregate their responses. This is expensive—instead, maintain a dedicated aggregation topology that pre-computes global windows.

Netflix’s Edge Gateway metrics system uses a hybrid approach: partition-local windows for per-service metrics (fast queries, no coordination), and a secondary global aggregation topology for cross-service dashboards. The global topology reduces 10,000 microservice streams into a single aggregated stream with tolerable latency (~5 seconds end-to-end).

Implementation Walkthrough

GitHub Link:

https://github.com/sysdr/sdc-java/tree/main/day47/day47-sliding-window-analytics

Step 1: Define Window Configuration

We implement multiple window sizes to serve different analytical needs. The 1-minute window detects immediate issues, 5-minute windows smooth noise, 15-minute windows identify sustained trends:

Duration oneMinWindow = Duration.ofMinutes(1);
Duration fiveMinWindow = Duration.ofMinutes(5);
Duration fifteenMinWindow = Duration.ofMinutes(15);
Duration hopInterval = Duration.ofSeconds(10);

TimeWindows oneMinHopping = TimeWindows
  .ofSizeWithNoGrace(oneMinWindow)
  .advanceBy(hopInterval);

Architectural Decision: No grace period initially—we prioritize deterministic window boundaries over late event handling. Production systems start here, then add grace periods based on observed late-arrival patterns.

Step 2: Build Windowed Aggregation Topology

The stream processing topology aggregates events into windowed state stores. Each log event contains metrics (error rate, latency, throughput) that we aggregate into moving averages:

streamsBuilder
  .stream("log-events")
  .groupByKey()
  .windowedBy(oneMinHopping)
  .aggregate(
    WindowStats::new,
    (key, event, stats) -> stats.update(event),
    Materialized.<String, WindowStats, WindowStore<Bytes, byte[]>>as("one-min-windows")
      .withValueSerde(windowStatsSerde)
  );

The materialized view name (”one-min-windows”) becomes the state store name for interactive queries. Kafka Streams automatically manages this store across multiple instances using partition assignment.

Step 3: Interactive Query API

The REST API exposes current moving averages by querying the underlying state stores. The critical challenge: state stores are partitioned—we need to discover which instance holds the data for a given key:

@GetMapping("/trends/{serviceId}")
public TrendResponse getTrends(@PathVariable String serviceId) {
  StreamsMetadata metadata = streams.metadataForKey(
    "one-min-windows", 
    serviceId, 
    Serdes.String().serializer()
  );
  
  if (metadata.equals(streams.localMetadata())) {
    return queryLocalStore(serviceId);
  } else {
    return forwardToInstance(metadata.host(), metadata.port(), serviceId);
  }
}

Step 4: Caching Layer for Query Performance

Querying RocksDB state stores on every API request creates I/O bottlenecks. We cache computed trends in Redis with short TTLs matching the hop interval:

String cacheKey = "trend:" + serviceId + ":" + System.currentTimeMillis() / hopMillis;
TrendResponse cached = redis.get(cacheKey);

if (cached != null) return cached;

TrendResponse computed = computeFromStateStore(serviceId);
redis.setex(cacheKey, hopSeconds, computed);
return computed;

Performance Impact: Cache hit rate >90% reduces state store queries by 10x, dropping p99 latency from 25ms to <3ms.

Working demo link :

Production Considerations

Memory Management: Monitor state store disk usage with kafka.streams.state.store.bytes.total. Each window instance is ~100 bytes, so 10K keys × 30 windows × 100 bytes = 30MB per window size. With three window sizes and replicas, expect ~180MB total.

Out-of-Order Events: Set grace periods to p95 network latency (typically 100-500ms for same-region). Monitor kafka.streams.late.record.drop.total to detect excessive late arrivals indicating misconfigured grace periods or upstream delays.

Query Latency: p99 query latency should stay <10ms for local queries, <50ms for remote instance queries. High latency indicates either state store compaction issues or insufficient instance resources.

Failure Scenarios:

• Instance crash: State restores from changelog (~10-30 seconds for moderate state), queries fail until restoration completes

• Network partition: Queries to unreachable instances timeout, implement circuit breakers with 3-second timeouts

• Slow consumers: Kafka consumer lag increases, windows compute with stale data—monitor records-lag-max

Scaling to Production

Uber’s ride request monitoring processes 100K+ events/second using 20 Kafka Streams instances, each managing ~5K unique keys. They partition by geographic region (rider location hash) to enable local aggregations and regional dashboards. Their sliding windows use 30-second hops, creating 10 overlapping windows per 5-minute interval.

Key scaling insights from their architecture:

• State store size grows with unique key count, not event volume

• Hop size determines computational cost—10-second hops cost 6x more than 60-second hops

• Query latency depends on instance locality—co-locate API and stream processing when possible

[ Subscribe Now & Save 50% ]

What We’re Building Today

Today we implement production-grade time-based windowing for real-time log analytics:

• Tumbling Windows: Fixed-size, non-overlapping time windows for discrete period aggregations

• Hopping Windows: Overlapping time windows for trend detection with configurable advance intervals

• Session Windows: Dynamic windows based on activity gaps for user session analytics

• Windowed Metrics Engine: Real-time calculation of count, sum, average, min, max per window

• Late Data Handling: Grace periods and watermark management for out-of-order events

• Window State Persistence: RocksDB-backed state stores with changelog topics for fault tolerance

• Interactive Queries: REST API exposing current and historical window results in real-time

System processes 50,000+ events/second with sub-100ms window computation latency and maintains exactly-once window semantics even during failures.

Why This Matters: The Foundation of Real-Time Analytics

Every production monitoring system, business intelligence dashboard, and real-time alerting platform relies on time-based windowing. When Netflix monitors video quality metrics per 5-minute window across 200+ million users, when Uber calculates surge pricing based on 1-minute ride request windows per geographic area, or when Amazon tracks order volumes in 15-minute windows for capacity planning—they all use the same fundamental windowing patterns we’re implementing today.

The challenge isn’t just aggregating data over time—it’s handling late-arriving events, managing state for millions of concurrent windows, ensuring exactly-once semantics despite failures, and providing low-latency access to both current and historical window results. Window boundaries create consistency challenges: should an event timestamped at 10:59:59 but arriving at 11:00:01 belong to the 10:00-11:00 window or be discarded? How long do you wait for stragglers before finalizing a window?

Modern stream processing platforms solve these problems through watermarks (tracking event time progress), grace periods (allowing late data within bounds), and stateful processing (maintaining window state across crashes). Understanding these patterns transforms you from writing batch aggregation scripts to building the real-time analytics engines that power modern data-driven companies.

Read more

Today we’re implementing a production-grade MapReduce framework for batch log analysis:

• Distributed MapReduce Engine: Complete map-shuffle-reduce pipeline processing millions of log events

• Word Count & Pattern Analysis: Real-time pattern frequency detection across distributed log streams

• Fault-Tolerant Task Scheduling: Coordinator-worker architecture with automatic task retry and failure recovery

• Scalable Storage Backend: Partitioned intermediate results with efficient shuffle operations

Why This Matters: The Foundation of Big Data Processing

While Kafka Streams excels at real-time processing, many analytics workloads require batch processing of historical data. MapReduce remains the fundamental pattern behind modern data processing frameworks like Apache Spark, Hadoop, and even cloud-native services like AWS EMR and Google Dataflow.

When Netflix analyses viewing patterns across billions of log events to optimize content recommendations, when Uber processes trip data to identify demand hotspots, or when Amazon analyses customer behaviour across terabytes of clickstream data—they’re all using MapReduce-style distributed processing. The pattern we implement today scales from processing megabytes on your laptop to petabytes across thousands of machines.

The key insight: MapReduce transforms complex distributed data processing into two simple operations (map and reduce) while hiding the complexity of data distribution, parallel execution, fault tolerance, and result aggregation. This abstraction enables data engineers to focus on business logic while the framework handles distributed systems complexity.

System Design Deep Dive: MapReduce Architecture Patterns

1. Map-Shuffle-Reduce Pipeline Architecture

MapReduce divides computation into three distinct phases:

Map Phase: Each mapper processes a subset of input data independently, emitting key-value pairs. For log analysis, mappers extract patterns like error codes, user IDs, or URL paths from log entries. The critical design decision is data partitioning—how you split input data determines parallelism and load balance.

Shuffle Phase: The framework groups all values with the same key and routes them to the appropriate reducer. This is where network I/O becomes the bottleneck. Production implementations use combiners (local reducers) to minimize data transfer. Twitter’s implementation processes 100TB+ daily logs, reducing shuffle data by 80% through combiner optimization.

Reduce Phase: Each reducer aggregates values for its assigned keys. Reducers must handle partial failures—if a reducer crashes mid-processing, the framework restarts it with the same input data. This requires idempotent operations.

Trade-off: MapReduce optimizes for throughput over latency. While Kafka Streams provides sub-second processing, MapReduce batch jobs might take minutes or hours. Choose MapReduce when you need to process complete datasets with strong consistency guarantees over time-sensitive results.

2. Coordinator-Worker Task Scheduling

The coordinator (master) maintains the distributed system state:

• Task Assignment: Assigns map and reduce tasks to available workers

• Progress Tracking: Monitors task completion and detects stragglers

• Failure Detection: Identifies crashed workers and reschedules their tasks

• Data Locality: Preferentially assigns tasks to workers with local data access

The CAP Theorem Implication: Our coordinator becomes a single point of failure, choosing consistency (CP) over availability (AP). In production, systems like Google’s MapReduce use Chubby (distributed lock service) or Apache ZooKeeper to make the coordinator highly available. For our implementation, we accept this trade-off for simplicity.

Straggler Mitigation: LinkedIn’s MapReduce jobs process 40PB monthly. They discovered that 10% of tasks take 3x longer than average (stragglers). The solution: speculative execution—launch backup tasks for slow-running jobs and use whichever completes first.

3. Partitioned Intermediate Storage

Between map and reduce phases, intermediate results must be stored and shuffled:

Disk-Based Storage: Mappers write output to local disk partitioned by reduce key. This provides fault tolerance—if a reducer fails, intermediate data persists for retry. The trade-off is I/O overhead.

In-Memory Optimization: Modern implementations like Apache Spark cache intermediate data in memory when possible, achieving 10-100x speedup. We implement a hybrid approach—memory buffers with disk spillover.

Hash Partitioning: We use consistent hashing to distribute keys across reducers. This ensures even load distribution and enables horizontal scaling. Amazon’s internal MapReduce processes 100M+ keys per second using murmur3 hash with 10,000 reduce partitions.

4. Fault Tolerance Through Task Retry

Distributed systems fail constantly at scale. Google’s cluster of 10,000 machines experiences:

• 20 machine failures per day

• 1000 hard drive failures per year

• Network partitions several times per week

Our MapReduce framework implements three fault-tolerance mechanisms:

Heartbeat-Based Failure Detection: Workers send periodic heartbeats to the coordinator. Missing 3 consecutive heartbeats triggers task rescheduling. This detects crashes, network partitions, and hung processes.

Task-Level Idempotency: Each task produces deterministic output for the same input. If a task executes twice (due to retry), the final result remains correct. This requires careful handling of side effects.

Partial Result Recovery: If 95% of map tasks complete but 5% fail, we only retry the failed tasks rather than restarting the entire job. This dramatically improves completion time for large jobs.

5. Backpressure and Resource Management

Without proper backpressure, the system floods:

• Memory Exhaustion: Fast mappers overwhelm slow reducers, filling intermediate storage

• Network Saturation: Shuffle phase consumes all bandwidth, starving other cluster traffic

• Disk Thrashing: Too many concurrent writes cause random I/O patterns

Our implementation uses:

• Task Throttling: Limit concurrent map tasks based on available worker memory

• Flow Control: Reducers signal backpressure when input buffers reach 80% capacity

• Resource Quotas: Each job gets CPU/memory/disk quotas to prevent resource starvation

Uber’s MapReduce platform processes 100PB+ daily. They implement hierarchical fair scheduling—giving priority queues 60% of cluster resources while ensuring batch jobs get at least 20%.

Implementation Walkthrough: Building the Framework

GitHub Link :

https://github.com/sysdr/sdc-java/tree/main/day45/mapreduce-log-processor

Core Components Architecture

Our system comprises five microservices:

MapReduce Coordinator: Spring Boot service managing job lifecycle, task scheduling, and failure recovery. Exposes REST API for job submission and status queries. Maintains task state in PostgreSQL for fault tolerance.

Map Worker Pool: Horizontally scalable workers consuming log batches from Kafka, applying user-defined map functions, and writing partitioned intermediate results to Redis. Each worker processes 10,000 events/second with automatic retry on transient failures.

Reduce Worker Pool: Workers reading shuffled data from Redis, applying reduce functions, and persisting final results to PostgreSQL. Implements combiner pattern to minimize network transfer during shuffle phase.

Storage Layer: Redis stores intermediate map outputs with 1-hour TTL. PostgreSQL persists final results with proper indexing for analytical queries. Kafka provides input log stream with replay capability for job reruns.

API Gateway: Rate-limited REST endpoints for job submission, progress monitoring, and result retrieval. Implements circuit breaker pattern to prevent cascade failures.

Implementation Flow

1. Job Submission Phase:

@PostMapping("/jobs")
public JobStatus submitJob(@RequestBody JobRequest request) {
    // Validate user-defined map/reduce functions
    validateUserCode(request.getMapFunction(), request.getReduceFunction());
    
    // Create job metadata and initial tasks
    Job job = jobRepository.save(new Job(request));
    createMapTasks(job, request.getInputTopic(), request.getNumMappers());
    
    // Publish job to task queue for worker pickup
    coordinatorService.scheduleJob(job);
    return new JobStatus(job.getId(), "RUNNING");
}

2. Map Phase Execution:

@KafkaListener(topics = "map-tasks")
public void executeMapTask(MapTask task) {
    try {
        // Consume log batch from Kafka with offset management
        List<LogEvent> logs = kafkaConsumer.poll(task.getPartition());
        
        // Apply user map function: log -> List<KeyValue>
        List<KeyValue> mappedResults = logs.stream()
            .flatMap(log -> mapFunction.apply(log))
            .collect(Collectors.toList());
        
        // Partition by reduce key and write to Redis
        Map<Integer, List<KeyValue>> partitions = 
            partitionByKey(mappedResults, task.getNumReducers());
        
        partitions.forEach((partition, data) -> 
            redisTemplate.opsForList()
                .rightPushAll(partitionKey(task.getJobId(), partition), data)
        );
        
        // Report completion to coordinator
        coordinatorService.completeTask(task.getId());
    } catch (Exception e) {
        coordinatorService.failTask(task.getId(), e);
    }
}

3. Shuffle and Reduce Phase:

@Scheduled(fixedDelay = 1000)
public void executeReduceTasks() {
    List<ReduceTask> tasks = coordinatorService.getReadyReduceTasks();
    
    tasks.parallelStream().forEach(task -> {
        // Fetch all values for assigned partition from Redis
        List<KeyValue> partitionData = redisTemplate.opsForList()
            .range(partitionKey(task.getJobId(), task.getPartition()), 0, -1);
        
        // Group by key and apply reduce function
        Map<String, List<String>> grouped = partitionData.stream()
            .collect(Collectors.groupingBy(
                KeyValue::getKey,
                Collectors.mapping(KeyValue::getValue, Collectors.toList())
            ));
        
        List<Result> results = grouped.entrySet().stream()
            .map(e -> new Result(e.getKey(), reduceFunction.apply(e.getValue())))
            .collect(Collectors.toList());
        
        // Persist final results to PostgreSQL
        resultRepository.saveAll(results);
        coordinatorService.completeTask(task.getId());
    });
}

Working demo link :

Key Architectural Decisions

Why Redis for Intermediate Storage: We need fast random writes (map output) and sequential reads (reduce input). Redis provides 100K+ ops/second with built-in persistence. The alternative (disk-only) reduces throughput by 10x but improves fault tolerance.

Task Granularity: Each map task processes 10,000 log events. Smaller tasks increase scheduling overhead; larger tasks reduce parallelism. This aligns with Google’s MapReduce guideline: task execution time should be 1-10 minutes.

Heartbeat Interval: Workers send heartbeats every 5 seconds with 15-second timeout. Faster intervals waste network bandwidth; slower intervals delay failure detection. This matches AWS EMR’s production settings.

Production Considerations

Performance Characteristics: Our framework processes 50,000 events/second with 4 map workers and 2 reduce workers. Horizontal scaling is linear up to 20 workers (200K events/sec) before coordinator bottleneck. Memory footprint: 2GB per worker for 100K intermediate key-value pairs.

Monitoring Strategy: Track critical metrics:

• Job completion rate and average duration

• Task failure rate by type (map vs reduce)

• Shuffle data volume (indicates skew problems)

• Worker CPU/memory/disk utilization

• Coordinator queue depth (scheduling bottleneck indicator)

Failure Scenarios:

• Worker Crash: Coordinator detects via heartbeat timeout, reschedules in-progress tasks

• Coordinator Crash: New coordinator reads job state from PostgreSQL, resumes scheduling

• Data Skew: One reduce key has 80% of data—causes straggler. Solution: implement combiner or split hot keys

• Network Partition: Workers isolated from coordinator. Solution: implement split-brain detection with fencing tokens

Scalability Bottlenecks: The coordinator handles 1000 tasks/second. Beyond that, implement hierarchical coordinators or consistent hashing for task assignment. Redis shuffle layer supports 1M keys before requiring Redis Cluster (sharding).

Scale Connection: MapReduce in Production Systems

Google’s original MapReduce processed 20PB per day across 1000s of machines. Modern implementations scale further:

Facebook’s Corona: Schedules 100,000+ MapReduce jobs daily across 60,000 machines, processing 600PB of data monthly. They implement three-level scheduling hierarchy to scale the coordinator.

LinkedIn’s Hadoop: Runs 250,000 jobs per day with average job completion time of 4 minutes. Their optimization: aggressive speculative execution reduces tail latency by 40%.

Twitter’s Scalding: Processes 100TB+ daily logs for real-time and batch analytics. They combine MapReduce (batch) with Storm (streaming) for lambda architecture.

The pattern we implemented today—map/shuffle/reduce with fault-tolerant coordination—remains the foundation of modern big data processing, evolved into frameworks like Spark and Flink but retaining the same core abstractions.

• Live metrics aggregation system processing 40,000+ events/second with sub-second latency

• Kafka Streams processor performing windowed aggregations, percentile calculations, and anomaly detection

• Real-time dashboard API serving live statistics with WebSocket updates

• Production monitoring stack with Grafana dashboards tracking stream processor health

• Fault-tolerant state management using RocksDB-backed state stores with changelog topics

Why This Matters: Observability at Internet Scale

When Netflix processes 450 billion events per day from their streaming platform, or Uber analyzes 100 million trip events daily, they need real-time visibility into system behavior. Traditional batch processing creates blind spots—by the time you see yesterday’s metrics, today’s incidents have already cascaded. Real-time stream processing transforms raw events into actionable insights within milliseconds, enabling immediate detection of anomalies, capacity issues, and user-impacting problems.

The challenge isn’t just aggregating data—it’s maintaining accurate state across failures, handling late-arriving events, managing memory with billions of unique keys, and providing consistent results during rebalances. A poorly designed streaming pipeline can lose data during crashes, produce duplicate counts after restarts, or fall behind during traffic spikes. Today we’ll build a production-grade monitoring system that handles these challenges using Kafka Streams’ exactly-once semantics, fault-tolerant state stores, and windowed aggregations.

System Design Deep Dive: Stream Processing Patterns

1. Stateful Stream Processing with RocksDB

Kafka Streams maintains state locally in embedded RocksDB databases, backed by Kafka changelog topics. When your stream processor calculates “requests per minute” or “95th percentile latency,” it’s not querying a database—it’s updating in-memory/on-disk state stores that survive process crashes. Each state store has a corresponding changelog topic that captures every state mutation. If a processor crashes, the replacement reads the changelog to rebuild state from the last checkpoint.

Trade-off: Local state provides sub-millisecond query latency but limits scalability to disk capacity per instance. For aggregations tracking millions of unique keys, you must partition state across multiple processor instances. The group_id determines which processor owns which keys—consistent hashing ensures the same keys always route to the same partition.

Anti-pattern: Storing unbounded state. Without time-based eviction (windowing) or size limits, state stores grow indefinitely. A service tracking “unique users per endpoint” will eventually run out of disk if it never expires old keys. Always implement retention policies aligned with business requirements.

2. Windowed Aggregations and Time Semantics

Stream processing must handle time ambiguity—events arrive with: processing time (when the processor sees it), event time (when it actually occurred), and ingestion time (when Kafka received it). A log event from 10 minutes ago arriving now (due to network delays) must contribute to the correct time window, not the current one.

Kafka Streams supports tumbling windows (non-overlapping 1-minute buckets), hopping windows (overlapping 5-minute windows advancing every 1 minute), and session windows (grouped by inactivity gaps). Each creates different trade-offs: tumbling windows provide simple counts but miss trends across boundaries; hopping windows smooth outliers but duplicate event processing; session windows handle bursty traffic but complicate memory management.

Production insight: Always configure grace periods for late arrivals. A 1-minute window with 30-second grace accepts events up to 1:30 after window close, balancing completeness against latency. LinkedIn’s Samza learned this the hard way—their initial streaming pipelines dropped 2% of events during peak load because they closed windows too aggressively.

3. Materialized Views and Interactive Queries

State stores serve dual purposes: internal processing state and queryable materialized views. Your Kafka Streams application can expose REST endpoints that query local state stores directly, bypassing external databases. When the dashboard requests “current requests/sec by endpoint,” the API queries the stream processor’s state store—no database roundtrip needed.

Scaling consideration: State stores are partitioned—a query for /api/users might land on instance-1, but that instance only holds state for partition 0. You need either: (1) scatter-gather queries across all instances, (2) routing proxy directing queries to correct partition, or (3) global state stores replicated to all instances. Global stores solve routing but triple memory usage for frequently queried data.

Twitter’s architecture: Their real-time analytics use interactive queries against Kafka Streams state stores for the first 7 days of data, then fall back to Druid for historical analysis. This hybrid approach balances query latency (5ms from state stores vs 50ms from Druid) against storage costs.

4. Exactly-Once Stream Processing

Kafka Streams achieves exactly-once semantics through transactional writes—each processing step (read input, update state, write output) executes atomically. If the processor crashes mid-transaction, the entire operation rolls back. This prevents duplicate counts after restarts, a common bug in at-least-once processing.

Implementation: Enable processing.guarantee=exactly_once_v2 and ensure all state operations happen within the topology. External side effects (database writes, API calls) break exactly-once guarantees—if your stream processor writes to PostgreSQL, then crashes, the Kafka message will be reprocessed but the DB write won’t roll back, creating duplicates.

Trade-off: Exactly-once processing adds 10-15% latency overhead from transactional commits. For monitoring dashboards where occasional duplicates are acceptable, at-least-once processing provides better throughput. For financial transactions or user account state, exactly-once is mandatory.

5. Stream Processing Failure Modes

Stream processors fail differently than request-response services. A crashed processor doesn’t just stop responding—it triggers rebalances that temporarily halt all partition processing. During rebalance, the dashboard shows stale data until state restoration completes (reading changelog topics can take 30-60 seconds for large state stores).

Cascading failures: One slow processor instance causes Kafka consumer group heartbeat timeouts, triggering rebalances across all instances, pausing all processing during state restoration, creating backlog that overloads instances when they resume. This cascade can bring down entire streaming pipelines.

Mitigation: Implement backpressure handling—if state stores can’t keep up with ingestion rate, the processor should pause consumption rather than accepting unbounded backlog. Configure max.poll.interval.ms generously (5 minutes) to prevent false-positive timeouts during legitimate processing spikes.

Implementation Walkthrough: Building the Monitoring Pipeline

GitHub Link:

https://github.com/sysdr/sdc-java/tree/main/day44/realtime-monitoring-dashboard 

Service Architecture

Our system consists of four Spring Boot services:

log-producer generates realistic log events (HTTP requests, database queries, cache operations) at 40,000 events/second. Each event includes timestamp, endpoint, response time, status code, and user identifier. Events flow into log-events Kafka topic with 12 partitions for parallelism.

stream-processor consumes from log-events, performs windowed aggregations (requests per minute, error rates, latency percentiles), and materializes results to state stores. It exposes REST endpoints querying these state stores—no external database required. The processor computes:

• Request count per endpoint per minute (tumbling window)

• Error rate by status code per 5-minute window (hopping)

• P50, P95, P99 latency using t-digest algorithm

• Anomaly detection flagging 3-sigma deviations

dashboard-api serves WebSocket connections, polling the stream processor’s interactive queries every second and pushing updates to connected clients. It maintains connection state in Redis for horizontal scaling—multiple API instances can serve different dashboard clients.

dashboard-ui provides a single-page React application with real-time charts. We use Recharts for time-series visualization and WebSocket API for live data streaming.

Kafka Streams Topology

StreamsBuilder builder = new StreamsBuilder();

KStream<String, LogEvent> events = builder.stream("log-events");

// Windowed aggregation: requests per endpoint per minute
KTable<Windowed<String>, Long> requestCounts = events
    .groupBy((key, event) -> event.getEndpoint())
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1)))
    .count(Materialized.as("request-counts-store"));

// Error rate calculation
KTable<Windowed<String>, Double> errorRates = events
    .groupBy((key, event) -> event.getEndpoint())
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
    .aggregate(
        ErrorRateAccumulator::new,
        (key, event, accumulator) -> accumulator.add(event),
        Materialized.as("error-rates-store")
    )
    .mapValues(acc -> acc.calculateRate());

The topology defines data flow transformations—grouping, windowing, aggregation—that execute across multiple instances. State stores (request-counts-store) automatically partition across stream processor instances based on endpoint hash.

Interactive Query Implementation

The stream processor exposes REST endpoints that query local state stores:

@GetMapping("/metrics/requests")
public Map<String, Long> getRequestCounts() {
    ReadOnlyWindowStore<String, Long> store = 
        streams.store(StoreQueryParameters.fromNameAndType(
            "request-counts-store", 
            QueryableStoreTypes.windowStore()
        ));
    
    Instant now = Instant.now();
    Instant start = now.minus(Duration.ofMinutes(5));
    
    Map<String, Long> results = new HashMap<>();
    store.all().forEachRemaining(kv -> {
        if (kv.key.window().end() >= start.toEpochMilli()) {
            results.put(kv.key.key(), kv.value);
        }
    });
    return results;
}

This query runs in O(k) time where k is the number of endpoints—no database query, no network calls, pure local state access.

Working Demo Link :

State Store Failure Recovery

When a stream processor crashes, Kafka Streams handles recovery automatically:

1. Consumer group rebalances—partitions reassign to surviving instances

2. New partition owner reads changelog topic from last committed offset

3. State store rebuilds from changelog (may take 30-60 seconds)

4. Processing resumes once state restoration completes

During restoration, the state store is unavailable—queries return 503. The dashboard API must handle this gracefully, showing “loading” state until the processor recovers.

Production Considerations

Performance bottlenecks: State store compaction is CPU-intensive—RocksDB background threads can consume 20-30% CPU even during steady-state operation. Monitor rocksdb.total-sst-files-size metric—growth beyond available disk indicates insufficient compaction. Increase rocksdb.max-background-compactions if you see compaction delays.

Memory management: Each windowed aggregation creates new state store entries—1,000 endpoints × 5-minute window retention × 12 windows = 60,000 state entries. With 40,000 events/sec, memory usage grows at ~500MB/hour. Implement time-based retention using suppress operators to purge old windows automatically.

Monitoring critical metrics:

• kafka-streams-state-store-lag: Indicates how far behind state stores are from Kafka topics (target: <1000)

• stream-processor-commit-latency-avg: Time to commit state changes (target: <100ms)

• rebalance-time: Downtime during partition reassignment (target: <30 seconds)

Failure scenario testing: Simulate instance crashes with kill -9, verify state restoration completes within SLA, confirm no data loss or duplicate counts. Test backpressure handling—what happens when ingestion rate exceeds processing capacity? The system should pause consumption rather than falling behind indefinitely.

Scale Connection: Real-World Stream Processing

LinkedIn’s Venice: Processes 400,000 events/second using Kafka Streams with 2TB of state distributed across 50 stream processor instances. They achieve P99 query latency of 5ms by pre-aggregating hot keys (top 1000 endpoints) into separate state stores cached in memory.

Uber’s AresDB: Real-time analytics on 100 million trips/day use GPU-accelerated aggregations in Kafka Streams pipelines. By offloading percentile calculations to CUDA kernels, they reduced per-event processing time from 2ms to 0.3ms, enabling real-time fraud detection across 10,000+ cities.

Netflix’s Keystone: Monitors 450 billion events/day from streaming devices using Kafka Streams for real-time alerting. They partition state by region (us-east-1, eu-west-1) to isolate failures—an outage in one region doesn’t halt global monitoring.

Key Insight: Real-time monitoring isn’t about speed—it’s about maintaining accurate state across failures while processing unbounded data streams. The hard problem is making your aggregations survive the chaos of production: crashes, rebalances, network partitions, and traffic spikes.

Today we’re implementing a production-grade state management system using Kafka log compaction:

• Compacted Topics that maintain only the latest state for each entity key

• State Producer Service generating entity lifecycle events with proper keying

• State Consumer Service maintaining current entity snapshots from the compacted log

• State Query API providing fast lookups of current entity state with Redis caching

Why This Matters: The State Management Challenge at Scale

Every distributed system faces the same fundamental challenge: how do you maintain current state across dozens of microservices without creating a monolithic database bottleneck?

Traditional approaches fail at scale. Storing complete event histories consumes unbounded storage. Database-per-service patterns create consistency nightmares during failures. Cache invalidation becomes impossibly complex with hundreds of service instances.

Log compaction solves this by treating your event log as a self-maintaining state store. Instead of storing every state transition, Kafka automatically retains only the latest value for each key. This gives you the benefits of event sourcing (complete audit trail, replayability, temporal queries) while maintaining bounded storage and fast state reconstruction.

Netflix uses this pattern to maintain current device registration state across 200+ million subscribers. When a device registers, deregisters, or updates settings, those events flow through compacted topics. Any service can rebuild complete device state by consuming from offset 0, getting only current registrations. Uber applies the same pattern to driver location state, maintaining billions of location updates while keeping storage constant.

Read more

Today we implement exactly-once processing semantics in our Kafka-based log processing system, guaranteeing no duplicate message processing even during failures:

• Idempotent Kafka producers preventing duplicate writes on network retries

• Transactional message processing with atomic offset commits and database writes

• Deduplication layer using Redis for distributed idempotency keys

• State reconciliation service detecting and recovering from processing anomalies

• End-to-end exactly-once pipeline from producer through consumer to database

Why This Matters: The $10 Million Double-Charge Problem

In 2019, a major payment processor experienced a 47-second network partition during peak Black Friday traffic. Their Kafka consumers lost connections, reconnected, and reprocessed 180,000 payment authorization messages—charging customers twice. The cost: $10.3 million in refunds, regulatory fines, and customer service overhead.

The root cause wasn’t Kafka. It was the absence of exactly-once semantics. Without idempotent producers, network retries created duplicate messages. Without transactional consumers, offset commits happened before database writes, causing reprocessing on crashes. Without deduplication, the same payment ID was processed multiple times.

Exactly-once processing isn’t about theoretical correctness—it’s about financial accuracy, compliance requirements, and system reliability at scale. When Uber processes 100 million trip events daily, Stripe handles billions in transactions, or AWS Lambda processes trillions of invocations, “at-least-once with deduplication” becomes a critical architectural pattern.

Read more

Today we implement horizontal scalability for log processing through Kafka’s partitioning and consumer group mechanisms:

• Topic partitioning strategy with semantic key selection for log event distribution

• Consumer group coordination enabling automatic load balancing across multiple consumer instances

• Dynamic rebalancing protocols handling consumer failures and scale-out scenarios gracefully

• Partition assignment strategies optimizing throughput for different log processing workloads

Why This Matters: The Parallel Processing Foundation

When Netflix processes 500 billion events daily or Uber handles 14 million trips per day, single-threaded processing isn’t an option. Kafka’s partitioning model solves the fundamental distributed systems challenge: how do we process millions of messages per second while maintaining order guarantees where they matter and maximizing parallelism everywhere else?

The architectural decision between using a single partition (strong ordering, limited throughput) versus multiple partitions (high throughput, ordering within partitions only) defines your system’s scalability ceiling. Companies like LinkedIn process 7 trillion messages daily through Kafka precisely because partitioning enables horizontal scaling - adding more consumers linearly increases processing capacity without architectural changes.

Understanding partition assignment and consumer group coordination is critical for system design interviews. When asked “how would you process 1 million events per second?”, the answer involves this exact partitioning strategy with consumer groups, not faster machines.

Read more

Nothing teaches better than “Code in Action”.

• Learn AI Agents : Join the AI agent revolution before your competition does.

• Explore more hands-on courses https://systemdrd.com

• Lifetime Access : 4 hands on courses + full portal with Pro Max offer → link

What We’re Building Today

• Consumer group architecture with automatic partition assignment and rebalancing

• Offset management strategies implementing at-least-once and exactly-once semantics

• Multi-threaded processing pipeline with parallel log transformation and enrichment

• Dead letter queue pattern for poison pill messages and retry exhaustion handling

Why This Matters

Consumer implementation determines your system’s throughput, reliability, and operational complexity. While producers are relatively simple—fire and forget with acks—consumers manage offset commits, rebalancing coordination, and state consistency across failures. At Netflix, consumer lag spikes directly correlate with degraded user experience as recommendations stale. Uber’s geospatial processing consumers must maintain exactly-once semantics to prevent duplicate ride assignments. The difference between naive polling loops and production consumer patterns is the gap between prototypes that process 500 events/second and systems handling 500,000 events/second with zero data loss.

System Design Deep Dive

Read more

Today we implement production-grade Kafka producers that form the critical ingestion layer of our distributed log processing system:

• Multi-source log shippers that collect logs from applications, services, and infrastructure components

• High-throughput Kafka producers capable of handling 50,000+ events/second with sub-10ms latency

• Intelligent batching and compression strategies that optimize network utilization and reduce Kafka broker load

• Producer-side monitoring and observability with Prometheus metrics, latency histograms, and error tracking

Why This Matters: The $10M Question of Log Ingestion

When Twitter experienced cascading failures in 2016, engineers discovered their log ingestion pipeline had dropped 40% of critical error logs during the incident. The missing data cost them millions in debugging time and prevented root cause analysis. The culprit? Naive Kafka producers without proper backpressure handling, retry logic, or circuit breakers.

At Netflix scale (processing 500+ billion events daily), every millisecond of producer latency multiplies across thousands of services. A poorly configured producer can create backpressure that cascades through your entire microservices ecosystem, causing request timeouts, degraded user experience, and revenue loss. Getting producer configuration right isn’t optional—it’s the difference between a system that scales gracefully and one that collapses under load.

System Design Deep Dive

1. Producer Acknowledgment Semantics and Durability Trade-offs

Kafka producers offer three acknowledgment levels, each representing a different point on the CAP theorem spectrum:

acks=0 (Fire-and-Forget): Producer doesn’t wait for broker acknowledgment. Achieves maximum throughput (100,000+ msg/sec) but risks data loss if brokers fail. Use case: High-volume metrics where occasional loss is acceptable (think Uber’s GPS tracking where losing 0.1% of coordinates doesn’t impact route calculation).

acks=1 (Leader Acknowledgment): Producer waits for leader broker confirmation. Balances throughput (50,000+ msg/sec) with durability. Risk: Data loss if leader fails before replication. This is the sweet spot for most log ingestion scenarios—you get reasonable guarantees without sacrificing performance.

acks=all (Full Replication): Producer waits for all in-sync replicas. Guarantees no data loss but reduces throughput (10,000-20,000 msg/sec) and increases latency (20-50ms). Essential for financial transactions or audit logs where every event must be preserved.

The architectural insight: There’s no universal “best” setting. Netflix uses acks=1 for application logs but acks=all for billing events. Your producer configuration should match your data’s business value.

2. Batching, Linger, and the Throughput-Latency Trade-off

Individual message sends are network-inefficient—each requires a round trip to the broker. Kafka producers solve this through batching, controlled by two critical parameters:

batch.size: Maximum batch size in bytes (default 16KB). Larger batches improve throughput by amortizing network overhead across multiple messages. At 32KB batches, you can achieve 3x throughput compared to individual sends.

linger.ms: How long the producer waits to accumulate messages before sending. Setting linger.ms=10 means “wait up to 10ms to build a larger batch.” This is counterintuitive—adding latency improves throughput.

The trade-off: A producer with batch.size=32KB and linger.ms=0 achieves low latency (2-5ms) but moderate throughput (20,000 msg/sec). Setting linger.ms=10 increases latency to 12-15ms but achieves 60,000+ msg/sec through better batching.

Real-world application: Uber’s log ingestion uses adaptive linger—during high traffic (rush hour), linger.ms=5 because batches fill quickly. During low traffic (3 AM), linger.ms=50 ensures efficient batching despite low event rates.

3. Compression and Network Efficiency

Uncompressed logs consume massive bandwidth. A typical application log (200 bytes) contains repetitive structure—timestamps, log levels, class names. Compression algorithms exploit this repetition:

snappy: Fast compression (20-30% size reduction) with minimal CPU overhead. Compresses at 250+ MB/sec, making it ideal for high-volume scenarios where CPU is precious.

gzip: Better compression (40-50% reduction) but slower (80-100 MB/sec). Use when network bandwidth is the bottleneck, not CPU.

lz4: Fastest option (300+ MB/sec) with moderate compression (25-35% reduction). The default choice for most production systems.

At 50,000 events/sec with 200-byte messages, that’s 10 MB/sec uncompressed. With lz4 compression, you’re down to 7 MB/sec—a 30% reduction in cross-datacenter bandwidth costs. Over a year, this saves hundreds of thousands in AWS data transfer fees.

4. Idempotence and Exactly-Once Semantics

Network failures create a subtle problem: when a send times out, did the broker receive the message? Retrying might create duplicates. Not retrying risks data loss.

Kafka’s idempotent producer solves this by assigning each message a sequence number. If the broker receives duplicate sequence numbers from the same producer, it deduplicates automatically. Enable with enable.idempotence=true.

The cost: Idempotence requires acks=all and limits in-flight requests to 5. This reduces maximum throughput from 100,000+ to 30,000-40,000 msg/sec. But you get exactly-once semantics—critical for financial logs, user analytics, or any scenario where duplicates corrupt downstream processing.

Architectural decision: LinkedIn’s log pipeline uses idempotent producers for user activity events (preventing double-counting in analytics) but non-idempotent for debug logs where occasional duplicates are harmless and throughput matters more.

5. Circuit Breakers and Graceful Degradation

When Kafka brokers become unavailable, naive producers queue messages in memory until they run out of heap, causing OutOfMemory crashes. Production systems need circuit breaker patterns:

buffer.memory: Maximum memory for buffering unsent messages (default 32MB). When exhausted, send() calls block or throw exceptions.

max.block.ms: How long to block before throwing TimeoutException (default 60 seconds). Setting this to 5000ms prevents cascading failures—your producer fails fast instead of hanging threads.

Circuit breaker integration: When error rates exceed thresholds, open the circuit and stop accepting new logs. This prevents producer services from crashing and allows graceful recovery when Kafka returns.

Amazon’s CloudWatch log ingestion uses a three-tiered degradation strategy: (1) Under normal operation, send to Kafka with acks=1. (2) During Kafka brownout (high latency), switch to acks=0 for non-critical logs. (3) During Kafka blackout, write critical logs to local disk and replay when connectivity returns.

Implementation Guide: Kafka Log Producers — High-Throughput Log Ingestion

A practical guide to building a production-style log ingestion system with Apache Kafka and Spring Boot. You’ll implement multiple producer strategies (throughput vs. durability), a reactive gateway, and observability.

Github Link :

https://github.com/sysdr/sdc-java/tree/main/day39/kafka-log-producers

Prerequisites

• JDK 17+

• Docker and Docker Compose

• Maven 3.8+

• (Optional) Git to clone the repo

Architecture at a Glance

Clients → Log Gateway (WebFlux) → Application Log Shipper → Kafka (application-logs)
                              → Transaction Log Shipper  → Kafka (transaction-logs)
                                                         → PostgreSQL (outbox)
         Infrastructure Shipper (scheduled)               → Kafka (infrastructure-metrics)

Design choices:

ConcernApplication / Infra ShippersTransaction Shipperacks1 (leader ack, higher throughput)all (durability)IdempotenceOffOnOutboxNoYes (PostgreSQL)Rate limit50K/sec (token bucket)Not applied

Step 1: Project Layout and Docker

Create a root folder (e.g. kafka-log-producers) with:

kafka-log-producers/
├── docker-compose.yml      # Kafka, Zookeeper, PostgreSQL, Prometheus, Grafana
├── pom.xml                 # Parent POM with modules
├── application-log-shipper/
├── infrastructure-log-shipper/
├── transaction-log-shipper/
├── log-gateway/
├── monitoring/
│   ├── prometheus.yml
│   └── dashboards/
└── setup.sh                # Start infra + create topics

docker-compose.yml (minimal): include services for zookeeper, kafka (Confluent 7.5), postgres (port 5433 to avoid clashes), prometheus (9090), grafana (3000). Wire Kafka to Zookeeper and add healthchecks so app containers can depends_on: kafka: condition: service_healthy.

Step 2: Application Log Shipper — High-Throughput Producer

Goal: Ingest application logs over HTTP and publish to application-logs with batching and rate limiting.

2.1 Dependencies (pom.xml)
Spring Boot Starter Web, Spring Kafka, Lombok, Micrometer (Prometheus), Guava (RateLimiter).

2.2 Producer configuration

Use a @Configuration class that builds a ProducerFactory<String, LogEvent> and a KafkaTemplate<String, LogEvent>. Key settings:

• BOOTSTRAP_SERVERS_CONFIG from spring.kafka.bootstrap-servers

• KEY_SERIALIZER: StringSerializer, VALUE_SERIALIZER: JsonSerializer (for a LogEvent POJO)

• Throughput tuning: ACKS_CONFIG = "1", COMPRESSION_TYPE_CONFIG = "lz4", BATCH_SIZE_CONFIG = 32768, LINGER_MS_CONFIG = 10, BUFFER_MEMORY_CONFIG = 67108864

• Reliability: RETRIES_CONFIG = 3, DELIVERY_TIMEOUT_MS_CONFIG = 30000, REQUEST_TIMEOUT_MS_CONFIG = 15000

2.3 Log event model

Simple POJO: eventId, source, level, message, timestamp, serviceId, traceId, metadata (Map). Use Instant for timestamp.

2.4 Sending with rate limit and metrics

In a service that uses KafkaTemplate and Micrometer MeterRegistry:

• Create a Guava RateLimiter at 50,000 permits per second.

• For each send: if !rateLimiter.tryAcquire(Duration.ofMillis(100)), increment a producer.throttled counter and throw a custom RateLimitException (e.g. HTTP 429).

• Start a Timer.Sample, then kafkaTemplate.send("application-logs", event.getEventId(), event).

• In the whenComplete callback: stop the timer (e.g. kafka.producer.send.duration), and increment either kafka.producer.success or kafka.producer.error counters.

2.5 REST API

• POST /api/v1/logs/ingest: accept a JSON body, map to LogEvent, generate eventId (e.g. UUID) and timestamp if missing, call the send service. Return 202 with eventId or 429 when rate limited.

Step 3: Transaction Log Shipper — Exactly-Once with Outbox

Goal: Accept transaction events, store in a DB outbox, then publish to Kafka with an idempotent producer so you can replay from the outbox if needed.

3.1 Dependencies
Add Spring Data JPA, PostgreSQL driver, and Spring Kafka.

3.2 Producer configuration

Same structure as the application shipper, but:

• ACKS_CONFIG = "all"

• ENABLE_IDEMPOTENCE_CONFIG = true

• Keep compression and batching (e.g. same batch/linger/buffer as above).

3.3 Outbox entity (PostgreSQL)

Table transaction_outbox: id (PK), transactionId (unique), userId, amount, currency, status, createdAt, sentAt. Use @Entity and @Table(name = "transaction_outbox").

3.4 Transaction event POJO

Fields such as: transactionId, userId, type, amount, currency, timestamp.

3.5 Outbox + send flow

In a @Transactional method:

1. Save a new row in transaction_outbox with status = "PENDING".

2. Call kafkaTemplate.send("transaction-logs", event.getTransactionId(), event).

3. In whenComplete: on success, set status = "SENT" and sentAt = now(), then save the entity; on failure, increment a transactions.failed counter (and optionally keep status for retry).

Use transactionId as the Kafka key for ordering and idempotency.

3.6 REST

• POST /api/v1/transactions: body with userId, type, amount, currency; build TransactionEvent with generated transactionId and timestamp; call the transactional send service; return 202 with transactionId.

Step 4: Log Gateway (Reactive)

Goal: Single API for clients; gateway forwards to the appropriate shipper.

4.1 Dependencies
Spring Boot WebFlux, Spring WebFlux WebClient (no blocking WebMVC).

4.2 WebClient

Create a WebClient bean (e.g. WebClient.builder().build() or with base URL). Use service names as hostnames when running in Docker (e.g.

http://application-log-shipper:8081

,

http://transaction-log-shipper:8083

).

4.3 Routes

• POST /api/v1/logs → POST http://application-log-shipper:8081/api/v1/logs/ingest (forward body).

• POST /api/v1/transactions → POST http://transaction-log-shipper:8083/api/v1/transactions (forward body).

• GET /api/v1/health → return {"status":"UP"}.

Return Mono<Map<String, Object>> from the shipper responses so clients see the same shape (e.g. eventId or transactionId).

Step 5: Infrastructure Log Shipper (Scheduled Metrics)

Goal: Periodically generate fake metrics and publish to infrastructure-metrics.

Use the same producer pattern as the application shipper (acks=1, batching, LZ4). Add a @Scheduled(fixedRate = 1000) method that builds a small batch of events (e.g. CPU, memory, disk) with timestamps and sends them via KafkaTemplate. No HTTP API required; optional actuator for health.

Step 6: Topics and Startup

In setup.sh (or equivalent):

1. docker compose up -d.

2. Wait for Kafka to be ready (e.g. loop with kafka-broker-api-versions --bootstrap-server localhost:9092).

3. Create topics if not exists:

   • application-logs (3 partitions, replication 1)

   • infrastructure-metrics (3 partitions)

   • transaction-logs (3 partitions)

Expose a short summary: Gateway 8080, Application 8081, Infra 8082, Transaction 8083, Prometheus 9090, Grafana 3000.

Step 7: Observability

• Prometheus: Scrape actuator metrics from each Spring Boot app (e.g. /actuator/prometheus). Configure targets in prometheus.yml.

• Grafana: Add Prometheus as data source; import or create a dashboard for:

  • rate(kafka_producer_success_total[1m]) (throughput)

  • rate(kafka_producer_error_total[1m]) (errors)

  • kafka_producer_send_duration_seconds (latency)

  • producer_throttled_total (rate limiting)

Verification

1. Health: curl http://localhost:8080/api/v1/health

2. Log: curl -X POST http://localhost:8080/api/v1/logs -H "Content-Type: application/json" -d '{"source":"test","level":"INFO","message":"Hello"}' → expect 202 and an eventId.

3. Transaction: curl -X POST http://localhost:8080/api/v1/transactions -H "Content-Type: application/json" -d '{"userId":"u1","type":"PAYMENT","amount":99.99,"currency":"USD"}' → expect 202 and transactionId.

4. Kafka: kafka-console-consumer --bootstrap-server localhost:9092 --topic application-logs --from-beginning --max-messages 5 (and similarly for transaction-logs, infrastructure-metrics).

5. Rate limit: Send a burst of requests (e.g. script or load test); confirm 429s and producer.throttled in Prometheus.

Production Considerations

Performance Characteristics and Bottlenecks

In our load tests, a single producer instance achieves 50,000+ events/sec with the following resource profile:

• CPU: 2 cores at 60% utilization (mostly spent on JSON serialization and lz4 compression)

• Memory: 512MB heap (200MB for buffering, 300MB for Spring Boot overhead)

• Network: 7-10 MB/sec outbound (with compression), 1 MB/sec inbound (acknowledgments)

The bottleneck shifts based on configuration. With acks=all, network latency becomes the constraint (limited by round-trip time to replicas). With acks=0, CPU becomes the constraint (serialization can’t keep up). With compression disabled, network bandwidth saturates first.

Failure Scenarios and Recovery

Kafka Broker Failure: Producers automatically retry failed sends (up to 3 attempts) with exponential backoff. After retries exhaust, events are written to a dead letter queue for manual investigation. We alert when DLQ depth exceeds 1000 events.

Producer Service Crash: Unsent messages in memory are lost. This is acceptable for logs but unacceptable for transactions—hence the transaction shipper’s use of transactional outbox pattern with PostgreSQL.

Network Partition: Circuit breakers open after 10 consecutive failures, preventing memory exhaustion. Producers enter degraded mode, writing critical logs to local disk for later replay. We monitor circuit breaker state changes—sustained open state indicates infrastructure issues.

Monitoring and Alerting Strategy

We implement three-tier alerting:

• P0 (Page immediately): Producer error rate > 5% for 5 minutes (indicates Kafka cluster issues)

• P1 (Alert during business hours): Average latency > 100ms (indicates broker saturation)

• P2 (Daily summary): DLQ depth > 100 events (indicates intermittent serialization errors)

Dashboards show producer throughput, latency histograms, batch size distributions, and error rates across all shipper types. This gives operators real-time visibility into ingestion health.

Scale Connection: Producer Patterns at FAANG

Netflix: Runs 100,000+ producer instances across their microservices, each configured with adaptive batching that adjusts linger.ms based on traffic patterns. Their producers include custom interceptors that sample 1% of logs for trace analysis while sending 100% to Kafka—achieving observability without overwhelming their trace backends.

Uber: Implemented geographic producer routing—log events from EU riders go to EU Kafka clusters, reducing cross-datacenter latency from 150ms to 5ms. They use asynchronous sends with callback handlers that update per-datacenter success metrics, enabling rapid detection of regional Kafka issues.

Airbnb: Uses priority-based producers where booking logs get acks=all and idempotence while search logs use acks=0. This heterogeneous configuration optimizes for both data criticality and throughput, running 40,000+ events/sec on shared infrastructure.

Working Code Demo:

Today you’ll deploy a production-grade distributed log streaming platform that forms the backbone of modern event-driven architectures. By the end of this lesson, you’ll have:

• Multi-broker Kafka cluster with 3 nodes configured for high availability and fault tolerance

• Partitioned topic architecture supporting parallel log processing at 50,000+ events per second

• Comprehensive monitoring stack with real-time metrics for throughput, latency, and consumer lag

• Automated health checking system that validates cluster state and triggers alerts on degradation

• Load testing framework simulating production traffic patterns with configurable throughput profiles

Read more

A production-grade priority-based log processing system that ensures critical messages bypass normal processing queues:

• Multi-tier message routing with 4 priority levels (CRITICAL, HIGH, NORMAL, LOW)

• Dedicated consumer pools for high-priority logs with 10x faster processing SLAs

• Priority escalation engine that auto-promotes aged messages to prevent starvation

• Comprehensive monitoring tracking queue depths, processing latency, and priority distribution

Why This Matters: The 3AM Wake-Up Call Problem

When payment processing fails at Stripe, security breaches occur at AWS, or fraud detection triggers at PayPal, these critical events can’t wait behind millions of routine info logs. A single delayed security alert can mean the difference between detecting a breach in minutes versus hours.

In 2021, a major cloud provider experienced a 4-hour outage partly because critical infrastructure alerts were buried in normal log queues. Their monitoring system generated 500,000 events per second, but the 12 critical alerts indicating cascading failures were delayed by 18+ minutes in standard FIFO processing. Priority queues solve this by guaranteeing sub-second processing for critical events regardless of overall system load.

At scale, this pattern becomes essential: Uber processes 50,000 fraud detection events per second with <100ms SLAs while normal trip logs can tolerate 5-second latencies. Netflix routes critical streaming failures through dedicated high-priority paths while routine engagement logs use standard queues. The architectural challenge is implementing priority without creating starvation, head-of-line blocking, or resource exhaustion from priority escalation.

Read more

• Dead Letter Queue infrastructure for capturing and analyzing failed log processing attempts with automatic retry mechanisms

• Poison message detection system that prevents infinite retry loops and isolates problematic events

• DLQ monitoring dashboard with Grafana visualizations showing failure patterns, retry metrics, and message inspection capabilities

• Reprocessing pipeline enabling manual intervention and automated recovery from transient failures

Why This Matters: The Hidden Cost of Message Failures

When Uber processes 15 billion location updates daily, even a 0.01% failure rate means 1.5 million lost events. Without dead letter queues, these failures cascade: poison messages block consumer threads, retries amplify database load, and critical data vanishes silently. Netflix discovered this during a payment processing incident where failed transactions entered retry loops, creating 50x database load that took down their entire billing system for 3 hours.

Dead letter queues solve three production-critical problems: they prevent poison messages from blocking healthy traffic, preserve failed events for forensic analysis, and enable controlled reprocessing without impacting live systems. Amazon’s order processing uses DLQs to quarantine corrupted events that would otherwise trigger cascading failures across inventory, payment, and shipping services.

System Design Deep Dive

Pattern 1: Dead Letter Exchange with Message TTL

The foundation of DLQ architecture combines message expiration with automatic rerouting. When a consumer fails to process a message after exhausting retries, Kafka routes it to a dedicated dead letter topic rather than discarding it. This pattern preserves message ordering guarantees while preventing head-of-line blocking.

Trade-off Analysis: Immediate DLQ routing reduces consumer latency but may discard recoverable failures. Delayed routing with exponential backoff (1s, 2s, 4s, 8s) handles transient errors but increases system complexity. Production systems typically use hybrid approaches: fast-fail for validation errors, delayed retry for network timeouts.

Scalability Bottleneck: DLQs can become write-heavy during cascading failures. A database outage causing 10,000 msg/sec to fail creates a DLQ spike that overwhelms monitoring systems. Solution: Rate-limit DLQ writes and aggregate failure metrics.

Pattern 2: Poison Message Detection

Poison messages—events that consistently fail processing despite retries—create infinite loops that waste resources. Detection requires tracking per-message failure counts and identifying retry patterns. The pattern: attach metadata (attempt_count, first_failure_timestamp, error_signature) to each message, increment counters on failure, and route to DLQ after threshold breach.

Critical Insight: Hash-based error signatures prevent different exceptions from counting toward the same threshold. A message failing with “DatabaseTimeout” then “ValidationError” represents two distinct failure modes requiring separate analysis. Netflix uses error signature clustering to identify systemic issues vs. data-specific problems.

Anti-pattern: Global retry counters that reset on consumer restart. This creates a vulnerability where poison messages repeatedly block processing after each deployment. Solution: Store retry state in persistent headers, not consumer memory.

Pattern 3: Graduated Retry Strategy

Not all failures are equal. Network timeouts justify aggressive retries; schema validation errors don’t. Graduated retry implements failure classification: transient errors (network, rate limits) get exponential backoff up to 10 attempts, permanent errors (validation, authorization) go straight to DLQ, and ambiguous errors (generic exceptions) get limited retries.

Implementation Complexity: Distinguishing transient from permanent failures requires exception taxonomy. Spring Boot’s @Retryable annotation supports this but needs careful configuration. The challenge: third-party library exceptions that don’t clearly indicate failure type.

Production Example: Uber’s trip processing retries location errors (GPS drift, network issues) but immediately DLQs invalid passenger IDs. This prevents wasting resources on unrecoverable errors while maximizing success rate for transient issues.

Pattern 4: DLQ Monitoring and Alerting

Effective DLQ systems require real-time visibility into failure patterns. Key metrics: DLQ ingestion rate (msgs/sec), failure type distribution (validation vs. timeout vs. business logic), message age in DLQ, and reprocessing success rate. Alerts trigger when: DLQ rate exceeds 5% of main topic volume, any single error type dominates (>50% of failures), or messages remain in DLQ beyond retention policy.

Observability Challenge: Correlating DLQ messages back to original requests for distributed tracing. Solution: Preserve correlation IDs through the entire retry chain, enabling reconstruction of the complete processing timeline.

Alert Fatigue Prevention: Aggregate similar failures into single alerts rather than individual message notifications. A schema change causing 1000 validation failures should generate one “schema mismatch” alert, not 1000.

Pattern 5: Controlled Reprocessing Pipeline

Dead letter queues aren’t graveyards—they’re holding areas for recovery. Reprocessing requires: manual inspection tools for root cause analysis, fix-and-replay mechanisms for code bugs, and automated recovery for transient failures that resolved. The pattern uses a separate reprocessing consumer that reads from DLQ at controlled rates, applies fixes/transformations, and publishes to original topic.

Consistency Consideration: Reprocessed messages may arrive out-of-order relative to newer events. For log processing, this is acceptable (logs are idempotent). For financial transactions, it’s catastrophic. Solution: Timestamp-based deduplication and order verification before committing reprocessed events.

Capacity Planning: Reprocessing after major incidents can create thundering herd. If 100,000 messages accumulated during a database outage, naive replay at full speed overwhelms the now-healthy database. Solution: Rate-limited replay with circuit breakers.

Github Link:

https://github.com/sysdr/sdc-java/tree/main/day36/dlq-log-system

Implementation Walkthrough

Core Architecture

Our implementation creates three Kafka topics: log-events (main processing), log-events-retry (temporary retry holding), and log-events-dlq (permanent failures). The consumer uses Spring Kafka’s error handler with custom retry logic that routes messages through this topology.

Step 1: Enhanced Consumer with Retry Semantics

The LogConsumerService adds retry metadata to message headers before reprocessing. Each failure increments retry-count, records error-type, and timestamps the failure. After 3 attempts (configurable), messages route to DLQ with full diagnostic context. This design ensures forensic data survives through the failure chain.

Architectural Decision: We use Kafka headers for retry state rather than external storage (Redis/database) because headers travel with the message, preventing state synchronization issues during consumer scaling or rebalancing.

Step 2: DLQ Producer with Classification

The DeadLetterQueueService receives failed messages and classifies them by error type: VALIDATION, TIMEOUT, PROCESSING, UNKNOWN. Classification drives alerting and reprocessing strategies. Validation errors likely need code fixes, timeouts might self-resolve, processing errors require case-by-case analysis.

Implementation Detail: Error classification uses pattern matching on exception stack traces. This is brittle but pragmatic—proper exception hierarchies across microservices is ideal but rarely achievable in practice.

Step 3: Monitoring Integration

Micrometer counters track DLQ metrics by error type: dlq.messages.total{type=VALIDATION}, dlq.messages.total{type=TIMEOUT}. Prometheus scrapes these and Grafana dashboards visualize failure patterns. We also emit custom metrics for average time-to-DLQ (how quickly failures are detected) and DLQ processing lag (how far behind the reprocessing consumer is).

Step 4: Reprocessing API

The API Gateway exposes endpoints for DLQ operations: list failed messages with filters, inspect individual message details including full payload and error history, manually trigger reprocessing with optional transformations, and bulk replay with rate limiting. This transforms DLQ from black hole to operational tool.

Testing Failure Scenarios

The integration test suite simulates production failure modes: poison messages with invalid JSON, database timeouts during peak load, network partitions during message processing, and schema evolution mismatches. Each test verifies that messages reach DLQ with correct classification and that retries respect exponential backoff timings.

Load Test Design: We inject 1% failure rate into 10,000 msg/sec load to validate that DLQ handling doesn’t impact healthy message throughput. Success criteria: main topic latency remains under 100ms p99 despite DLQ activity.

Working Code Demo:

Production Considerations

Performance Profile: DLQ operations add 2-5ms latency per failed message (header writes, topic routing). At 0.1% failure rate (100 failures/sec in a 100K msg/sec system), DLQ overhead is negligible. But during cascading failures, DLQ writes can spike to 50K msg/sec, requiring dedicated partitions and consumer groups.

Monitoring Strategy: Alert on DLQ rate > 5% of main topic volume (indicates systemic issue), messages in DLQ older than 24 hours (retention violation), reprocessing success rate < 80% (fix-and-replay not working), and DLQ consumer lag growing (reprocessing falling behind).

Failure Mode: DLQ itself can fail. When DLQ topic is unavailable, consumers must decide: drop messages (data loss), block processing (availability loss), or buffer locally (memory exhaustion). Our implementation uses local disk buffering with size limits, preferring temporary degradation over silent data loss.

Capacity Planning: Size DLQ topic for 10x average failure rate to handle spike scenarios. If normal failure rate is 100 msg/sec, provision DLQ for 1000 msg/sec sustained. DLQ retention should match investigation SLAs—24 hours minimum, 7 days recommended.

Connection to Scale: FAANG DLQ Patterns

Netflix’s payment processing uses multi-tier DLQs: immediate DLQ for validation errors, 3-hour delayed queue for transient failures, 24-hour manual review queue for business logic errors. This graduated approach maximizes automatic recovery while minimizing human intervention. Their DLQ dashboards show real-time failure taxonomy, enabling rapid incident response.

Amazon’s order processing DLQs preserve idempotency keys and business context, allowing customer service to manually retry failed orders without duplicate charges. During Black Friday, their DLQ systems capture millions of failures without impacting checkout latency, then automatically reprocess overnight as infrastructure stabilizes.

Next Steps

Tomorrow we implement priority queues, enabling critical logs to bypass normal processing delays and reach consumers within milliseconds.

Today we implement intelligent routing mechanisms that direct different log types to specialized processing pipelines:

• Content-based routing engine that inspects log attributes and routes to appropriate Kafka topics

• Multiple specialized consumer pipelines (security, performance, application, system logs)

• Dynamic routing rules supporting regex patterns and severity-based filtering

• Fanout patterns for logs requiring multiple processing paths simultaneously

Why This Matters: The Routing Challenge at Scale

When Uber processes 100 billion log events daily, they don’t send every log through the same pipeline. Security incidents need immediate alerting within milliseconds, performance metrics aggregate into time-series databases, application errors route to incident tracking systems, and audit logs archive to long-term storage. Each pipeline has different latency requirements, storage patterns, and processing logic.

Without intelligent routing, you face two critical problems: resource waste (processing irrelevant logs consumes compute unnecessarily) and latency inflation (high-priority security events queue behind low-priority debug logs). Netflix learned this lesson during a critical security incident when P0 alerts drowned in millions of debug logs, delaying detection by 18 minutes. Their solution? Topic-based routing that isolated security logs into dedicated high-priority pipelines.

The architectural challenge isn’t just filtering - it’s building routing logic that scales to millions of events per second while maintaining low latency, supports dynamic rule updates without deployment, and handles the fanout complexity when single events need multiple processing paths.

System Design Deep Dive

Pattern 1: Content-Based Routing with Topic Segmentation

Traditional log processing systems use a single queue, creating head-of-line blocking where high-priority logs wait behind low-priority ones. Topic-based routing solves this by inspecting message content and directing to specialized topics.

Architecture Decision: Use Kafka topic segmentation rather than application-level filtering. When a log arrives, the routing service examines attributes (severity, source, type) and publishes to specific topics: logs-security, logs-performance, logs-application, logs-system. Each topic has independent consumer groups with different processing characteristics.

Trade-off Analysis: Content inspection adds 2-3ms latency at the router. However, specialized pipelines reduce downstream processing time by 10-100x by eliminating irrelevant log filtering at each consumer. For high-throughput systems (>10K events/sec), the routing overhead is negligible compared to gains from targeted processing.

Failure Mode: Routing logic bugs can silently drop critical logs. Mitigation: implement a catch-all default topic and monitor routing decision metrics. If 90% of logs route to default, your routing rules are failing.

Pattern 2: Dynamic Routing Rules Engine

Hard-coded routing logic requires deployment for rule changes. Production systems need runtime rule updates for emergency response (route all logs from compromised service to security pipeline immediately).

Implementation: Store routing rules in Redis with pattern matching using regex and composite conditions. Router loads rules at startup and subscribes to Redis pub/sub for updates. Each rule defines: match pattern (source service, severity level, contains keyword), destination topic, priority (when multiple rules match).

Rule Example:
- Pattern: severity=ERROR AND service=payment-api
- Destination: logs-critical-business
- Priority: 1 (highest)

Scalability Consideration: Pattern matching is CPU-intensive. At 50K events/sec, regex evaluation can become the bottleneck. Solution: compile regex patterns once at rule load, use simple string comparisons for common cases (severity levels), and implement rule caching for repeated patterns.

Pattern 3: Multi-Destination Fanout

Some logs need multiple processing paths simultaneously. Security logs might need real-time alerting AND compliance archival AND audit trail storage. Single-destination routing creates duplication complexity.

Kafka Approach: Publish to multiple topics atomically. The routing service evaluates all rules, collects matching destinations, and sends the log to each topic in a single transaction. Kafka’s producer batching optimizes multi-topic writes.

Critical Implementation Detail: Use Kafka transactions to ensure all-or-nothing delivery. If routing to 3 topics, either all 3 succeed or none do. Partial failures create data inconsistency across pipelines.

CAP Theorem Implication: Fanout increases write latency (waiting for multiple topic acks). For 3-way fanout with min.insync.replicas=2, you need 6 broker acknowledgments. This favors consistency over latency. For latency-critical paths, consider async fanout with best-effort delivery to secondary topics.

Pattern 4: Priority-Based Topic Allocation

Not all logs are equal. Security incidents need immediate processing; debug logs can tolerate delays. Separate topics enable independent consumer scaling and resource allocation.

Resource Mapping:

• logs-critical: 16 partitions, 8 consumer instances, dedicated CPU/memory

• logs-high: 12 partitions, 4 consumer instances

• logs-medium: 8 partitions, 2 consumer instances

• logs-low: 4 partitions, 1 consumer instance

Auto-Scaling Strategy: Monitor consumer lag per topic. When logs-critical lag exceeds 100 messages, scale consumers within 30 seconds. Low-priority topics can tolerate minutes of lag before scaling.

Back-Pressure Handling: When downstream processing can’t keep up, routing logic can implement adaptive throttling - temporarily route low-priority logs to batch processing while maintaining real-time flow for critical logs.

Pattern 5: Routing Metrics and Observability

Routing decisions are invisible in single-queue systems. Topic-based routing enables granular observability: how many logs per source service, severity distribution per topic, routing rule hit rates, and misrouted log detection.

Key Metrics:

• Routing decision latency (p50, p99, p999)

• Logs per topic per second (detect anomalies)

• Rule match rate (identify unused rules)

• Default topic rate (detect routing failures)

Alerting Strategy: If >5% of logs route to default topic, routing logic is failing. If critical topic receives >10x normal rate, investigate potential security incident or service failure. If routing latency p99 exceeds 10ms, router is becoming bottleneck.

Github Link:

https://github.com/sysdr/sdc-java/tree/main/day35/log-routing-system

Implementation Walkthrough

Routing Service Architecture

The routing service sits between log producers and Kafka topics. It receives logs via REST API, evaluates routing rules, and publishes to appropriate topics.

Core Components:

1. REST Controller: Accepts log events, validates format, returns 202 Accepted immediately

2. Routing Engine: Evaluates rules from Redis cache, determines destination topics

3. Kafka Producer: Publishes to multiple topics with transaction support

4. Rule Manager: Loads rules at startup, subscribes to Redis updates, recompiles patterns

Implementation Flow:

1. Log arrives at POST /api/logs
2. Validate JSON structure (reject invalid immediately)
3. Evaluate routing rules in priority order
4. Collect all matching destinations (fanout)
5. Begin Kafka transaction
6. Publish to each destination topic
7. Commit transaction (all-or-nothing)
8. Return 202 to client
9. Record routing metrics

Consumer Pipeline Specialization

Each topic has dedicated consumers optimized for their log type:

Security Pipeline (logs-security):

• Real-time processing (no batching)

• Immediate alerting to PagerDuty

• Enrichment with threat intelligence

• Storage in security SIEM

Performance Pipeline (logs-performance):

• Batched aggregation (10-second windows)

• Time-series database writes

• Percentile calculation

• Grafana dashboard updates

Application Pipeline (logs-application):

• Error tracking integration

• Stack trace analysis

• User session correlation

• Ticket creation for errors

Configuration-Driven Routing Rules

Rules defined in YAML, loaded to Redis:

rules:
  - name: security-critical
    priority: 1
    conditions:
      severity: [ERROR, FATAL]
      source: [auth-service, payment-api]
    destinations: [logs-security, logs-critical]
  
  - name: performance-metrics
    priority: 2
    conditions:
      type: metric
      metric_name: "response_time_*"
    destinations: [logs-performance]

The rule manager compiles these into efficient matching logic, caching compiled patterns for reuse.

Working Code Demo:

Production Considerations

Performance Characteristics: Routing adds 2-3ms p50 latency, 5-8ms p99. For 50K events/sec, router needs 4-8 cores with rule caching. Kafka producer batching reduces per-message overhead to <100μs.

Failure Scenarios:

• Rule compilation failure: Fall back to default routing, alert ops team

• Kafka topic unavailable: Queue in Redis with TTL, retry with backoff

• Transaction timeout: Reduce fanout destinations, implement async fallback

• Redis connection loss: Use last-known-good rule cache, alert for manual intervention

Monitoring Requirements:

• Track routing decision latency (should be <5ms p99)

• Monitor logs per topic per second (detect anomalies)

• Alert on default topic rate spikes (routing failures)

• Track transaction abort rate (downstream capacity issues)

• Consumer lag per topic (identify processing bottlenecks)

Capacity Planning: Each router instance handles 10-15K events/sec. For 100K events/sec, deploy 8-10 router instances behind load balancer. Each Kafka topic needs partitions equal to max consumer count for horizontal scaling.

Scale Connection: Enterprise Routing at FAANG

Netflix routes 500K events/sec across 50+ specialized topics. Their routing engine uses multi-stage filtering: cheap checks first (severity string comparison), expensive checks last (regex pattern matching). They implement circuit breakers per topic - if security topic consumers are down, route security logs to backup archival topic to prevent data loss.

Amazon’s CloudWatch Logs uses hierarchical routing with namespace isolation. Each AWS service has dedicated topic namespaces, enabling independent scaling and preventing noisy neighbor problems. Their routing SLA: 99.99% of logs routed correctly within 10ms.

Next Steps

Tomorrow we implement dead letter queues for handling logs that fail processing despite retries, completing our fault-tolerance architecture.

Today we implement reliable message processing with intelligent failure handling:

• Manual acknowledgment control to prevent message loss during processing failures

• Configurable retry mechanisms with exponential backoff for transient errors

• Dead letter queues to isolate poison pill messages from healthy traffic

• Idempotency tracking to guarantee exactly-once processing semantics

Why This Matters

In distributed systems, message acknowledgment determines whether your system loses data or processes it twice. At scale, the difference is billions of events. When Uber’s payment system processes ride completions, a lost acknowledgment means a driver doesn’t get paid. A duplicate acknowledgment means charging a rider twice. Netflix’s recommendation system processes 500 billion events daily—without proper acknowledgment strategies, their entire pipeline would grind to halt from poison pill messages or cascade into infinite retry loops.

The acknowledgment pattern defines your system’s reliability guarantees. Auto-commit provides at-most-once delivery (fast but lossy). Manual commit after processing gives at-least-once (reliable but potentially duplicate). Transactional commit achieves exactly-once (correct but complex). Production systems trade throughput for reliability based on business requirements—financial transactions need exactness, video view counts tolerate approximation.

Read more

• Consumer group architecture with automatic partition rebalancing across multiple instances

• Parallel log processing pipeline handling 50,000+ events/second with sub-100ms P99 latency

• Offset management strategies for reliable message consumption and failure recovery

• Backpressure-aware processing with dynamic batch sizing and flow control mechanisms

Why This Matters: The Consumer Scalability Challenge

While producers must handle bursts of incoming events, consumers face a different scaling challenge: processing throughput must match or exceed production rate to prevent unbounded queue growth. At Uber, their logging infrastructure processes 100 billion events daily across thousands of consumer instances. When a deployment temporarily doubles processing latency from 50ms to 100ms, consumer lag can balloon to hours of backlog within minutes, causing cascading failures in dependent systems that rely on near-real-time log insights.

The consumer side introduces unique distributed systems challenges that don’t exist for producers. Consumer groups must dynamically rebalance partition assignments as instances fail or scale, requiring consensus protocols that temporarily pause all consumption. Netflix’s consumer infrastructure restarts ~10,000 instances daily across their fleet, triggering ~500 rebalance operations per minute during peak deployment windows. Poor rebalancing strategies can create 30-60 second processing gaps, causing violations of their 99.9% SLA for anomaly detection pipelines that power their recommendation engine.

Read more