Day 70: Compliance Report Generation and Export Pipeline
What We’re Building Today
Audit Event Consumer: Kafka consumer that ingests and deduplicates compliance-relevant events into a dedicated audit store
Multi-Format Export Engine: Strategy-pattern exporters producing PDF, CSV, and JSON reports for regulatory consumption
Scheduled Report Generator: Cron-driven GDPR, SOC2, and PCI-DSS report automation with async generation and status tracking
Compliance Report API: REST endpoints for on-demand report generation, status polling, and authenticated file download
Why This Matters
Every company handling user data operates under compliance frameworks—GDPR, SOC2, PCI-DSS, HIPAA. When an auditor arrives (or a regulator sends a letter), the difference between a 3-hour response and a 3-week scramble is whether your compliance data is pre-collected and queryable or scattered across operational databases.
The naive approach—running ad-hoc SQL queries against your production database when audit season arrives—fails at scale. Uber processes 50M+ trips daily; their operational database can’t absorb analytical compliance queries. Netflix handles thousands of GDPR data-access requests monthly via dedicated compliance infrastructure. Stripe’s audit system is completely isolated from its payment processing path.
The pattern: stream audit events into a compliance-specific read model continuously, then generate reports asynchronously from that model. This decouples compliance data collection from report generation, and both from your operational systems.
System Design Deep Dive
1. Event Sourcing as Your Audit Foundation
Compliance reporting requires an immutable, append-only audit trail. Every user data access, deletion, authentication event, and admin action becomes an event with a unique ID, actor, timestamp, and outcome.
The critical property: audit events must be immutable after creation. This is non-negotiable for regulatory validity. Your audit store should never support UPDATE on event records—only INSERT and SELECT. Compliance frameworks treat mutable audit logs as evidence of tampering.
AuditEvent {
id: UUID (Kafka message key - used for deduplication)
userId, action, resourceType, resourceId
eventTime, ipAddress, outcome, metadata
serviceOrigin // which microservice emitted this
}Trade-off: Event sourcing means your compliance store grows indefinitely. Plan for partitioned tables by month and archival to cold storage after 7 years (the SOC2/GDPR retention floor).
2. CQRS: Separate Your Compliance Read Model
