Week 3 Integrated Logging Pipeline (MVP): From Serialization to Production-Style Observability
Software engineers rarely build “a logging feature” in isolation. In real systems you build a path: ingestion → parsing → normalization → validation → enrichment → output, with each stage designed to survive change (new formats, new schemas, new sources) without breaking downstream consumers.
This post is a production-oriented course recap of Week 3 (Data Serialization and Formats) and a practical guide for an integrated MVP that wires the key stages into one service you can run locally.
If you’re a developer, student, or system design engineer, the goal is simple:
Understand the “why” behind each lesson day (15–21)
See how the integrated service composes them
Ship a working pipeline you can test, operate, and extend
Agenda
What you’re building (and what “integrated” really means)
Week 3 course map (Days 15–21)
Core concepts (schemas, evolution, normalization, enrichment)
Step-by-step implementation guide (hands-on)
How to measure success (metrics you should track)
What to extend next (Day 16/17/19/20 integration plan)
What this integrated project builds
This project implements a single linear pipeline that mirrors a production-style log path:
INPUT → NORMALIZER → (optional validation) → ENRICHMENT → OUTPUT
In the MVP currently wired end-to-end, the service focuses on:
Day 15 (JSON structured logs) as the ingestion shape
Day 18 (normalization) to produce a canonical
NormalizedLogDay 21 (enrichment) to attach operational context (hostname, environment, version hints, timestamps)
The repo intentionally keeps legacy lesson code under legacy/ unchanged and uses thin adapters under libs/ to integrate it safely.
One real-world integration detail you’ll run into (and this MVP handles): two legacy trees share the same top-level Python package name src. The adapters activate the right legacy code lazily by adjusting import paths and clearing sys.modules entries so one process can run both stages.
