Today's Mission: Transform Your Search Into a Power Tool
Ever wondered how Amazon lets you filter products by brand, price range, customer rating, and shipping speed all at once? Today we're bringing that same multi-dimensional filtering power to your log processing system. By the end of this lesson, you'll have a faceted search engine that can simultaneously filter logs by service, severity level, time range, and custom metadata - just like the search systems powering major platforms.
What We're Building Today
High-Level Agenda:
✅ Multi-dimensional log filtering system with 5+ facet types
✅ Dynamic facet generation from log metadata with real-time counting
✅ Interactive search interface with Google Cloud Skills Boost styling
✅ Performance-optimized indexes achieving sub-100ms response times
✅ Production-ready architecture handling 1000+ logs/second
This builds directly on Day 54's SQL-like query parser, adding the visual and interactive layer that makes complex searches intuitive for operations teams.
Why Faceted Search Matters in Production
When your system generates millions of logs per hour, finding relevant information becomes like searching for a needle in a haystack. Traditional search requires knowing exactly what you're looking for. Faceted search flips this - it shows you what's available to explore.
Preparing for a distributed systems interview?
→Download the free Interview Pack
→ Subscribe now to access source code repository - 200 + coding lessons
Real-World Impact:
Netflix: Engineers filter video streaming logs by region, device type, and error category simultaneously
Uber: Operations teams slice ride logs by city, driver status, and trip duration ranges
Slack: Support teams filter message logs by workspace size, feature usage, and performance metrics
The magic happens when you can combine filters dynamically. Instead of writing complex queries, you click "Error Level: Critical" + "Service: Payment" + "Last 1 Hour" and instantly see exactly what you need.
Core Concept: Faceted Search Architecture
Faceted search transforms structured log data into searchable dimensions called "facets." Each facet represents a filterable attribute with pre-computed value counts.
The Three Pillars of Faceted Search
1. Facet Extractor Analyzes incoming logs to identify searchable dimensions:
# Example log entry creates these facets:
{
"service": "payment-api",
"level": "error",
"region": "us-west-2",
"response_time": "250ms"
}
# Generates facets:
# service: [payment-api, user-api, order-api]
# level: [error, warn, info, debug]
# region: [us-west-2, us-east-1, eu-west-1]
# response_time: [0-100ms, 100-500ms, 500ms+]
2. Facet Index Manager Maintains real-time counts for each facet value combination:
Service facets: payment-api (1,247), user-api (892), order-api (445)
Level facets: error (156), warn (234), info (2,198)
Combined facets: payment-api + error (89), user-api + warn (67)
3. Search Coordinator Orchestrates complex multi-facet queries across distributed indexes while maintaining response speed under 100ms.
Architecture Integration
Your faceted search system integrates into the distributed log processing pipeline as an interactive query layer:
Data Flow:
Log Ingestion → Structured logs enter the system
Facet Extraction → Automated metadata analysis identifies filterable dimensions
Index Updates → Real-time facet counting and aggregation
Search Interface → Interactive filters for complex query building
Result Rendering → Filtered log results with highlighted matches
This sits between your Day 54 query parser (which handles the actual search execution) and Day 56's real-time indexing (which will make facet updates instantaneous).
Implementation Highlights
Backend Architecture
Facet Engine Core : day55-faceted-search/backend/app/services/facet_engine.py
class FacetEngine:
def __init__(self, redis_url: str = "redis://localhost:6379", db_path: str = "data/logs.db"):
self.redis_client = redis.from_url(redis_url, decode_responses=True)
self.db_path = db_path
self.facet_definitions = {
'service': {'type': 'categorical', 'display': 'Service'},
'level': {'type': 'categorical', 'display': 'Log Level'},
'region': {'type': 'categorical', 'display': 'Region'},
'response_time_range': {'type': 'numeric', 'display': 'Response Time'},
'hour_of_day': {'type': 'temporal', 'display': 'Time of Day'}
}
self.init_database()
def init_database(self):
"""Initialize SQLite database for log storage"""
conn = sqlite3.connect(self.db_path)
conn.execute('''
CREATE TABLE IF NOT EXISTS logs (
id TEXT PRIMARY KEY,
timestamp TEXT,
service TEXT,
level TEXT,
message TEXT,
metadata TEXT,
source_ip TEXT,
request_id TEXT,
region TEXT,
response_time INTEGER
)
''')
conn.execute('CREATE INDEX IF NOT EXISTS idx_service ON logs(service)')
conn.execute('CREATE INDEX IF NOT EXISTS idx_level ON logs(level)')
conn.execute('CREATE INDEX IF NOT EXISTS idx_timestamp ON logs(timestamp)')
conn.commit()
conn.close()Search Service Integration
class SearchService:
def __init__(self, facet_engine: FacetEngine):
self.facet_engine = facet_engine
async def search(self, search_request: SearchRequest) -> SearchResponse:
"""Perform faceted search on logs"""
start_time = time.time()
# Get filtered logs
logs = await self.facet_engine.search_logs(search_request)
# Get facets with current filters applied
facets_summary = await self.facet_engine.get_facets(search_request.filters)
query_time = (time.time() - start_time) * 1000 # Convert to milliseconds
return SearchResponse(
logs=logs,
total_count=facets_summary.total_logs,
facets=[facet.dict() for facet in facets_summary.facets],
query_time_ms=round(query_time, 2),
applied_filters=search_request.filters
)Frontend React Components
Interactive Facet Panel :
day55-faceted-search/frontend/src/components/FacetPanel.js
import React from 'react';
const FacetPanel = ({ facets, selectedFilters, onFilterChange }) => {
const handleFacetValueToggle = (facetName, value) => {
const currentValues = selectedFilters[facetName] || [];
const newValues = currentValues.includes(value)
? currentValues.filter(v => v !== value)
: [...currentValues, value];
onFilterChange(facetName, newValues);
};
if (!facets || facets.length === 0) {
return <div className="loading">Loading filters...</div>;
}
return (
<div className="facet-panel">
{facets.map((facet) => (
<div key={facet.name} className="facet-group">
<div className="facet-title">{facet.display_name}</div>
{facet.values.slice(0, 10).map((facetValue) => (
<div
key={facetValue.value}
className={`facet-value ${facetValue.selected ? 'selected' : ''}`}
onClick={() => handleFacetValueToggle(facet.name, facetValue.value)}
>
<input
type="checkbox"
className="facet-checkbox"
checked={facetValue.selected}
onChange={() => {}} // Handled by onClick above
/>
<span className="facet-label">{facetValue.value}</span>
<span className="facet-count">{facetValue.count}</span>
</div>
))}
{facet.values.length > 10 && (
<div className="facet-more">
+{facet.values.length - 10} more
</div>
)}
</div>
))}
</div>
);
};
export default FacetPanel;Performance Optimizations
Memory Management
Facet Caching: Keep frequently accessed facet counts in Redis for sub-millisecond access
Index Strategy: Create composite indexes on commonly filtered combinations (service + level, region + timestamp)
Query Optimization: Combine log retrieval and facet counting in single database round-trip
Response Time Targets
Initial Search: < 100ms response time
Filter Updates: < 50ms for facet recalculation
Large Datasets: Maintain performance up to 1M+ logs
Horizontal Scaling Results
1 Consumer: 32 msg/sec
2 Consumers: 65 msg/sec
4 Consumers: 127 msg/sec
8 Consumers: 248 msg/sec
Scaling efficiency: 95%+ linear improvement
Github Link :
https://github.com/sysdr/course-p/tree/main/day55/day55-faceted-search
Source Code Repository :
git clone https://github.com/sysdr/course.git
checkout day55
cd day55/day55-faceted-search
./build.sh
./run.sh
.demo.sh🚀 Build, Test & Demo Guide
Quick Setup Commands
# Project initialization
mkdir day55-faceted-search && cd day55-faceted-search
mkdir -p {backend/{app,tests},frontend/{src,public},tests/{unit,integration},docker,data}
# Backend dependencies (Python 3.11+)
pip install fastapi==0.104.1 uvicorn==0.24.0 redis==5.0.1 \
pydantic==2.5.0 pytest==7.4.3 structlog==23.2.0
# Frontend dependencies (Node.js 18+)
npm install react@18.2.0 react-dom@18.2.0 axios@1.6.0 \
@mui/material@5.14.15 recharts@2.8.0
Core Implementation Structure
day55-faceted-search/
├── backend/
│ ├── app/
│ │ ├── api/ # FastAPI routes
│ │ ├── models/ # Pydantic models
│ │ ├── services/ # Facet engine & search
│ │ └── main.py # Application entry
│ └── tests/ # Backend tests
├── frontend/
│ ├── src/
│ │ ├── components/ # React components
│ │ ├── services/ # API integration
│ │
│ └── public/ # Static assets
└── docker/ # Container configuration
Step-by-Step Build Process
1. Backend Services Setup
# Start Redis for facet caching
redis-server --daemonize yes
# Initialize SQLite database with indexes
python backend/app/services/facet_engine.py init-db
# Start FastAPI backend
cd backend && python -m app.main
Expected Output:
INFO: Uvicorn running on http://0.0.0.0:8000
INFO: Application startup complete
✅ Facet engine initialized with 5 facet types
✅ SQLite database ready with optimized indexes2. Frontend Development Server
# Start React development server
cd frontend && npm startExpected Output:
Compiled successfully!
Local: http://localhost:3000
On Your Network: http://192.168.1.x:3000
✅ Frontend dashboard accessibleTesting & Verification
Backend API Testing
# Generate sample data
curl -X POST "http://localhost:8000/api/logs/generate?count=500"
# Test faceted search
curl -X POST "http://localhost:8000/api/search/" \
-H "Content-Type: application/json" \
-d '{
"query": "",
"filters": {"level": ["error"], "service": ["payment-api"]},
"limit": 10
}'Performance Load Testing
# Test concurrent search performance
python tests/load/test_concurrent_search.py
# Expected metrics:
# ✅ 95th percentile: < 100ms
# ✅ Concurrent users: 100+
# ✅ Memory usage: < 200MB
# ✅ Throughput: 65+ queries/second
Integration Testing
# Run comprehensive test suite
python -m pytest tests/ -v --cov=app --cov-report=html
# Expected results:
# ======================== 12 passed in 2.45s ========================
# Coverage: 92%🎬 Live Demo Experience
Interactive Dashboard Features
1. Real-Time Facet Filtering
Open http://localhost:3000
Click "Generate Sample Data (500 logs)" button
Use facet panels on left to filter by:
Service: payment-api, user-api, order-api
Level: error, warn, info, debug
Region: us-west-2, us-east-1, eu-west-1
Response Time: 0-100ms, 100-500ms, 500ms+
2. Multi-Dimensional Search
Select multiple filters simultaneously
Watch result counts update in real-time
Observe facet counts recalculate based on current filters
Use search box for text filtering combined with facets
3. Performance Monitoring
Stats bar shows: Total Logs, Query Time, Active Filters
Response times consistently under 100ms
Smooth interactions with instant visual feedback
Google Cloud Skills Boost UI Styling
The interface features:
Clean Material Design with subtle shadows and rounded corners
Responsive Grid Layout adapting to desktop and mobile
Professional Color Scheme using Google's blue (#1a73e8) primary palette
Interactive Elements with hover states and smooth transitions
Accessibility Features with proper contrast and keyboard navigation
Production Deployment
Docker Container Setup
# Build and deploy with Docker Compose
docker-compose up --build -d
# Verify all services running
docker-compose psContainer Status:
NAME STATUS PORTS
redis-1 Up 0.0.0.0:6379->6379/tcp
faceted-backend-1 Up 0.0.0.0:8000->8000/tcp
faceted-frontend-1 Up 0.0.0.0:3000->3000/tcpHealth Verification
# Backend health check
curl http://localhost:8000/health
# Response: {"status": "healthy", "service": "faceted-search"}
# Frontend accessibility
curl -I http://localhost:3000
# Response: HTTP/1.1 200 OK
# Search system stats
curl http://localhost:8000/api/search/stats | jq🎯 Assignment: E-Commerce Faceted Search
Mission: Build a faceted search system for an e-commerce platform's log analysis.
Requirements:
Extract facets from order processing logs (service, status, region, amount_range)
Create interactive filter interface with real-time counting
Support combined filters (service=payment AND status=failed AND region=us-west)
Demonstrate 1000+ logs/second facet update performance
Show drill-down patterns (region → country → city)
Success Metrics:
Filter combinations return results in <100ms
Facet counts update within 5 seconds of new logs
Interface handles 10+ simultaneous facet filters
Zero query errors under sustained load
Solution Approach:
Service identification:
user,inventory,paymentRouting keys:
payment.processor.error,inventory.stock.updateTopic patterns:
payment.*,inventory.*,user.activity.*Specialized processing: payment errors → security, inventory → analytics
Performance testing: sustained 1000 msg/sec with <10ms routing latency
Key Takeaways & Next Steps
What You've Accomplished Today
You've built a production-ready faceted search system that demonstrates patterns used by major tech companies:
✅ Multi-dimensional filtering across categorical, numeric, and temporal facets
✅ Real-time facet counting with Redis caching for sub-50ms updates
✅ Interactive React interface with Google Cloud Skills Boost styling
✅ Performance optimization achieving 100+ concurrent user support
✅ Horizontal scaling with 95%+ linear improvement efficiency
Integration Context
Yesterday's Foundation: SQL-like query parser provides the execution engine Today's Enhancement: Visual, multi-dimensional filtering interface
Tomorrow's Evolution: Real-time facet updates as logs arrive
Real-World Applications
The faceted search patterns you implemented today power operational dashboards at companies processing billions of events daily:
GitHub: Routes deployment logs to CI/CD dashboards while sending performance metrics to capacity planning
Airbnb: Separates user activity logs from infrastructure monitoring for targeted team access
Shopify: Filters order processing logs by merchant size, geography, and payment method for fraud detection
Understanding faceted search architecture prepares you for senior roles where you'll design systems making complex data accessible to entire organizations.
Next: Day 56 - Implement real-time indexing of incoming logs