Week 5: Message Queues for Log Processing
When Log Processing Becomes a Messaging Problem
Production backends rarely process logs synchronously inside the HTTP request that generated them. Under load, tying observability to the critical path increases tail latency, hides failures behind timeouts, and makes it impossible to absorb traffic spikes without dropping data. Message queues decouple ingestion from processing—and this Spring Boot application shows how that decoupling composes into a deployable system.
What We Built
A single Spring Boot 3.2 application (
Week5IntegratedApplication) hosting seven composable messaging modules: RabbitMQ topic routing, Kafka producer semantics, consumer groups, manual acknowledgment with idempotency, content-based fan-out, dead-letter classification, and priority-tier processingProfile-driven infrastructure toggles (
app.kafka.enabled,app.rabbitmq.enabled,app.redis.enabled) so the same JAR runs on a laptop with in-memory brokers or against a full Docker stackResilience4j circuit breakers on Kafka producers and log processing, Spring Retry on transient failures, and SQL-backed DLQ records that survive broker outages
An operations dashboard at
http://localhost:8080/
aggregating Micrometer counters, module metrics, and a one-click demo that exercises every pipeline
Why This Matters
At Netflix, Uber, or Datadog scale, log pipelines ingest billions of events daily. No team writes directly to Elasticsearch from a servlet thread. Producers publish durable events; brokers buffer them; consumers work at a sustainable rate with explicit reliability contracts—acknowledgments, retries, dead-letter routing, and priority tiers. This project implements those contracts in one JVM so you can trace how they interact rather than studying each pattern in isolation.
System Design Deep Dive
Topic exchange routing decouples producers from topology. The RabbitMQ module publishes Day31LogEvent instances through RabbitTemplate when brokers are enabled, or through Day31InMemoryBrokerService locally—simulating logs-critical, logs-processing, and logs-monitoring queues with routing keys that encode severity and category. Producers never hard-code consumer topology; bindings decide destination—a direct analogue to production systems that hide queue names behind exchange configuration.
Kafka producer tuning exposes reliability trade-offs. Day32KafkaProducerService offers sync, async, and fire-and-forget send paths guarded by a Resilience4j circuit breaker. Producer config sets acks=all and idempotent sends via Week5KafkaConfig. A custom Day32LogPartitioner routes high-severity events to dedicated partitions—an operational concern when you need ordered processing per tenant without starving the rest of the topic.
At-least-once delivery requires business-layer idempotency. Day34LogProcessingService combines Spring Retry with hybrid deduplication via Day34IdempotencyKeyRepository and optional Redis SETNX. Committing an offset or ACK only tells the broker you took responsibility once; deduplication is what makes retries safe.
DLQ design needs taxonomy, not a single bucket. Day36DeadLetterQueueService classifies failures as transient (retry topic), permanent (DLQ), or ambiguous (retry then DLQ), always persisting to Day36DlqRecordRepository before optional Kafka mirroring—so operators query failure state even when the cluster is down.
