Week 3: Data Serialization and Formats
When Every Log Speaks a Different Language
Your platform ingests logs from Kubernetes sidecars, legacy syslog daemons, mobile SDKs, and internal microservices—each emitting a different wire format. By the time an on-call engineer searches for a trace ID, the data has passed through three serializers, two schema versions, and a parser that silently dropped half the fields. Production observability breaks not because you lack volume, but because you never agreed on what a log is.
What We Built
JSON ingestion pipeline with JSON Schema validation (networknt), Bean Validation, H2 persistence, and Kafka dead-letter routing for rejected events
Protocol Buffers and Avro pipelines with build-time code generation, binary storage, correlation-ID indexing, and measurable payload size metrics
Format normalization service that detects and converts among TEXT, JSON, Protobuf, and Avro through a shared canonical log model
Embedded schema registry with versioned subjects, SHA-256 fingerprinting, backward/forward compatibility checks, and Resilience4j circuit breaking on registration storms
Why This Matters
At Datadog-scale ingestion, serialization choices are cost centers. JSON is debuggable but bloated; Protobuf cuts wire size; Avro carries schema evolution for long-lived Kafka topics. Netflix and Uber run mixed-format pipelines because no single serializer wins every trade-off. The anti-pattern: one global format with no registry, then discovering incompatible consumers months later during a deploy.
System Design Deep Dive
This system is a single Spring Boot application hosting seven cooperating modules, sharing H2 or PostgreSQL persistence, optional Kafka, Redis caching, and Micrometer telemetry. The architectural bet is canonical intermediate representation: every format handler parses into CanonicalLog, then serializes to the target format—avoiding N×N conversion matrices.
JSON ingestion applies Bean Validation plus JSON Schema. Invalid events return HTTP 422 and route to logs.dlq when Kafka is enabled. Protobuf tracks sizeBytes per event to quantify the JSON-vs-binary delta. Avro V1/V2 .avsc files demonstrate schema evolution with backward-compatible defaults. The embedded schema registry runs in-process with compatibility modes and payload validation by schema ID.
Multi-Format Log Platform Architecture Show the single Spring Boot JVM hosting JSON, Protobuf, Avro, normalization, schema registry, syslog/journald adapters, and enrichment modules. Depict shared H2/PostgreSQL, optional Kafka topics (logs, logs.dlq, avro-log-events, raw-logs, enriched-logs-complete/partial), Redis cache tier, and the Prometheus/Grafana observability stack on ports 9090/9091/3000.
Syslog/journald adapters normalize infrastructure logs into the same platform. Enrichment resolves hostname, datacenter, and service metadata via PostgreSQL + optional Redis, branching Kafka Streams output into complete versus partial topics. Trade-offs: embedded registries lack multi-cluster federation; format auto-detection adds latency—pass content-type hints in hot paths.
