Day 65: Field-Level Encryption for Sensitive Log Data
What We’re Building Today
AES-256-GCM encryption service for PII fields (email, SSN, credit cards) in log events
Key rotation system with versioned encryption keys stored in distributed cache
Transparent decryption layer with role-based access to encrypted fields
Performance-optimized encryption pipeline handling 50,000+ events/second
Why This Matters
When logs contain personally identifiable information (PII), compliance frameworks like GDPR, HIPAA, and PCI-DSS mandate encryption at rest and in transit. However, encrypting entire log streams creates operational nightmares: debugging becomes impossible, pattern analysis breaks, and search performance tanks. Field-level encryption solves this by selectively protecting sensitive fields while keeping the rest of the log searchable and analyzable.
Netflix processes billions of log events daily, many containing user IDs, IP addresses, and session tokens. Their approach: encrypt only the 5-10% of fields containing PII, keeping 90% of log data in plaintext for operational visibility. This hybrid strategy reduces encryption overhead by 85% while maintaining compliance. The challenge isn’t just encryption—it’s building a system that remains performant, debuggable, and operationally practical at scale.
System Design Deep Dive
Preparing for a distributed systems interview?
→A free welcome gift: Download the free Interview Pack
→52 FAANG Questions Drill cards Cheatsheets Vault - Get it here
→ Subscribe now to access source code repository - 200 + coding lessons
1. Encryption Strategy: Field-Level vs Full Payload
The Trade-off: Encrypting entire log events is simple but operationally devastating. Field-level encryption adds complexity but preserves observability.
Full Payload Encryption (the naive approach):
Pros: Simple implementation, guaranteed protection
Cons: Logs become opaque binary blobs, regex/grep fails, storage increases 40%, search requires full decryption
Field-Level Encryption (production approach):
Pros: Selective protection, searchable metadata, 10x faster queries
Cons: Complex key management, schema awareness required, potential for data leakage in related fields
Production Pattern: Use a field classification system. Tag fields as PUBLIC, INTERNAL, or PII during schema definition. Only PII fields get encrypted. This requires schema evolution support—when you add a new PII field, historical logs remain unencrypted for that field, so the decryption layer must handle missing encrypted data gracefully.
2. Key Management: Rotation Without Downtime
The CAP Theorem Reality: You cannot have consistent key distribution, always-available encryption, and partition-tolerant key storage simultaneously.
Most systems choose AP (Available + Partition-Tolerant) for encryption keys:
Keys cached in Redis with 5-minute TTL
Key rotation happens asynchronously
Old keys remain valid for 24 hours during rotation
Accept eventual consistency: 0.01% of events might use stale keys during rotation windows
Anti-pattern: Storing keys in a central database and fetching on every encryption. This creates a single point of failure and bottlenecks throughput at ~5,000 ops/sec. Instead, use versioned key caching:
Key Structure:
{
"keyId": "encryption-key-v47",
"algorithm": "AES-256-GCM",
"key": "base64-encoded-key",
"validFrom": "2025-02-01T00:00:00Z",
"validUntil": "2025-03-01T00:00:00Z"
}
Each encrypted field stores its keyId version. Decryption services fetch the appropriate key from cache, falling back to the key management service (AWS KMS, HashiCorp Vault) only on cache miss.
3. Encryption Performance: Batching and Async Processing
The Bottleneck: AES-256-GCM encryption on a single core processes ~200MB/sec. For a system handling 50,000 events/sec with 5 PII fields each (250,000 encryption ops/sec), naive synchronous encryption saturates at 12,000 events/sec on a 16-core machine.
Solution: Async encryption pipeline with batching:
Kafka consumer reads log events at 50K/sec
Classification stage identifies PII fields (CPU-bound, parallelizable)
Batch accumulator groups 100 events (reduces per-event overhead)
Parallel encryption workers (16 threads) encrypt batches
Kafka producer publishes encrypted events
This architecture achieves 50,000 events/sec because:
Batching reduces context switching overhead by 60%
Parallel workers saturate all CPU cores
Async processing prevents backpressure to upstream services
Trade-off: Batching adds 50-100ms latency. For real-time alerting on PII exposure, maintain a separate fast-path pipeline that processes high-priority events synchronously.
4. Decryption Access Control: Role-Based Field Visibility
The Problem: Not all engineers should decrypt all PII. Support engineers need email addresses for customer lookup but shouldn’t see SSNs. Compliance teams need audit trails of who decrypted what.
Pattern: Attribute-Based Access Control (ABAC) at the field level:
Access Policy:
{
"role": "support-engineer",
"allowedFields": ["user.email", "user.name"],
"deniedFields": ["user.ssn", "payment.cardNumber"],
"auditRequired": true
}
When a user queries logs:
Parse query for requested fields
Check user role against field-level policies
Decrypt only allowed encrypted fields
Redact denied fields (replace with
[REDACTED-SSN])Log decryption event to audit trail
Failure Mode: Policy changes don’t propagate instantly. Use a policy versioning system where each decryption request includes a policyVersion timestamp. Services cache policies for 60 seconds, then check for updates. During policy updates, some requests may use stale policies—accept this as a trade-off for availability.
5. Storage Optimization: Encrypted Field Indexing
The Problem: Encrypted fields can’t be indexed or searched. How do you find “all logs with email=user@example.com” when emails are encrypted?
Solution: Deterministic encryption for indexable fields:
Use HMAC-SHA256 to generate a searchable hash:
hmac(email) = "abc123"Store both the hash and AES-encrypted value
Queries search by hash, retrieve encrypted value, decrypt for display
Trade-off: Deterministic encryption is vulnerable to rainbow table attacks if the keyspace is small (e.g., email domains). Mitigate with domain-specific salts: use different HMAC keys for different email domains.
Implementation Walkthrough
GitHub Link:
https://github.com/sysdr/sdc-java-p/tree/main/day65/log-encryption-systemStep 1: Encryption Service with Key Versioning
The EncryptionService handles field-level encryption with automatic key rotation:
@Service
public class EncryptionService {
private final RedisTemplate<String, EncryptionKey> keyCache;
private final KeyManagementClient kmsClient;
public EncryptedField encrypt(String fieldName, String value) {
EncryptionKey key = getCurrentKey(fieldName);
Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding");
byte[] iv = generateIV(); // 12 bytes for GCM
cipher.init(Cipher.ENCRYPT_MODE, key.getSecretKey(), new GCMParameterSpec(128, iv));
byte[] encrypted = cipher.doFinal(value.getBytes(UTF_8));
return new EncryptedField(
fieldName,
Base64.encode(encrypted),
key.getKeyId(),
Base64.encode(iv)
);
}
}
Architectural Decision: Store the initialization vector (IV) with each encrypted field. GCM mode requires unique IVs per encryption to maintain security. This increases storage by 16 bytes per field but prevents IV reuse attacks.
Step 2: Kafka Consumer with Async Encryption
The consumer processes logs in batches, encrypting PII fields asynchronously:
@Service
public class LogEncryptionConsumer {
private final ExecutorService encryptionPool = Executors.newFixedThreadPool(16);
@KafkaListener(topics = "raw-logs")
public void processLogs(List<LogEvent> events) {
List<CompletableFuture<LogEvent>> futures = events.stream()
.map(event -> CompletableFuture.supplyAsync(
() -> encryptSensitiveFields(event),
encryptionPool
))
.toList();
CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]))
.thenAccept(v -> publishEncryptedLogs(
futures.stream().map(CompletableFuture::join).toList()
));
}
}
Why Async? Blocking encryption on 16 threads would cap throughput at 12K events/sec. Async processing with CompletableFuture allows the consumer thread to batch-fetch the next 100 events while encryption workers process the current batch, achieving 50K+ events/sec.
Step 3: Role-Based Decryption Layer
The query service decrypts fields based on user roles:
@Service
public class LogQueryService {
public List<LogEvent> queryLogs(LogQuery query, UserContext user) {
List<LogEvent> results = logRepository.findByQuery(query);
return results.stream()
.map(event -> decryptAllowedFields(event, user))
.toList();
}
private LogEvent decryptAllowedFields(LogEvent event, UserContext user) {
Set<String> allowedFields = policyService.getAllowedFields(user.getRole());
event.getEncryptedFields().forEach((fieldName, encryptedValue) -> {
if (allowedFields.contains(fieldName)) {
String decrypted = encryptionService.decrypt(encryptedValue);
event.setField(fieldName, decrypted);
auditService.logDecryption(user, fieldName, event.getId());
} else {
event.setField(fieldName, "[REDACTED]");
}
});
return event;
}
}
Key Insight: Decryption happens lazily at query time, not during storage. This allows policy changes to take effect immediately—revoking SSN access from a role instantly prevents future decryptions without re-encrypting stored data.
Production Considerations
Performance: Encryption adds 2-5ms per event. Under load, batch 100 events and parallelize encryption across 16 workers to maintain 50K events/sec throughput. Monitor encryption_latency_p99 metric—if it exceeds 10ms, scale worker pools horizontally.
Monitoring: Track key rotation failures with key_rotation_error_rate. A spike indicates KMS availability issues. Set up alerts for decryption_access_denied_rate—sudden increases suggest policy misconfigurations or potential security incidents.
Failure Scenarios:
KMS unavailable: Encryption falls back to last-known cached keys (valid for 24h). Logs continue flowing; alert ops team.
Key rotation mid-flight: Events encrypted with old keys remain decryptable. Decryption service tries current key first, falls back to previous 3 versions.
Corrupted encrypted data: Add HMAC signatures to detect tampering. Reject events with invalid signatures; alert security team.
Working Demo Link
Scale Connection
Uber’s approach: Encrypt rider/driver PII in trip logs but keep geolocation data plaintext. This enables fraud detection (pattern analysis on routes) while protecting identities. They use field-level encryption with geographic key sharding—US data uses US KMS keys, EU data uses EU KMS keys, ensuring GDPR compliance.
Amazon’s pattern: Encrypt customer PII in CloudWatch logs but maintain separate encrypted search indexes using deterministic encryption. Queries search the index by hashed email, retrieve encrypted log IDs, then batch-decrypt full logs. This hybrid approach achieves sub-second search on billions of encrypted logs.
Next Steps
Tomorrow: Implement log redaction for compliance—automatically detect and redact SSNs, credit cards, and API keys in unstructured log text using regex patterns and ML-based PII detection.