Day 174: Create A/B Test Analysis Framework
Week 25: Business Analytics Use Cases | 254-Day Hands-On System Design Series
What We Build Today
Ingest experiment event logs from the distributed pipeline
Assign users deterministically to control/treatment groups via hash-based bucketing
Compute statistical significance (z-test, Welch’s t-test) on conversion metrics
Surface live results through a React dashboard with confidence intervals and lift %
Auto-register every experiment that appears in the log stream
Why A/B Testing Lives in the Log Pipeline
When Booking.com runs 1,000 simultaneous experiments, the raw signal is log data—page views, clicks, bookings—streaming through their event pipeline. Attaching the analysis framework directly to that pipeline means zero ETL delay: results update as logs arrive, not the next morning after a batch job.
Our log processor already tags every event with user_id, session_id, timestamp, and event_type. We add two fields: experiment_id and variant. The rest is statistics.
Core Concepts
Deterministic Assignment Users must land in the same variant on every visit. A simple hash solves this: hash(user_id + experiment_id) % 100 < split_pct → treatment. No database lookup, no race condition, same result every time.
Metric Aggregation Per variant we track impressions, conversions, and revenue inside a time-windowed accumulator. The window matches the experiment’s active period so stale events don’t pollute results.
Statistical Testing Two-sample z-test for proportions (conversion rate) and Welch’s t-test for continuous metrics (revenue per user). p-value and 95% confidence interval update every 30 seconds. When p < 0.05 and the CI excludes zero, the result is significant.
Sequential vs Fixed-Horizon Fixed-horizon tests enforce a minimum sample size before results are readable—blocking the peeking bias that inflates false-positive rates. Sequential testing (Airbnb’s method) uses alpha-spending to let teams check daily without inflating error. We support both modes via a config flag.
Architecture
Log Stream ──► Experiment Router ──► Metric Aggregator
│ │
Assignment Store Stats Engine (every 30s)
│
Result Store (SQLite)
│
FastAPI REST Layer (:8174)
│
React Dashboard