Today's Mission: Transform Your Search Into a Power Tool

Ever wondered how Amazon lets you filter products by brand, price range, customer rating, and shipping speed all at once? Today we're bringing that same multi-dimensional filtering power to your log processing system. By the end of this lesson, you'll have a faceted search engine that can simultaneously filter logs by service, severity level, time range, and custom metadata - just like the search systems powering major platforms.

What We're Building Today

High-Level Agenda:

✅ Multi-dimensional log filtering system with 5+ facet types
✅ Dynamic facet generation from log metadata with real-time counting
✅ Interactive search interface with Google Cloud Skills Boost styling
✅ Performance-optimized indexes achieving sub-100ms response times
✅ Production-ready architecture handling 1000+ logs/second

This builds directly on Day 54's SQL-like query parser, adding the visual and interactive layer that makes complex searches intuitive for operations teams.

Why Faceted Search Matters in Production

When your system generates millions of logs per hour, finding relevant information becomes like searching for a needle in a haystack. Traditional search requires knowing exactly what you're looking for. Faceted search flips this - it shows you what's available to explore.

Preparing for a distributed systems interview?

→Download the free Interview Pack

→ Subscribe now to access source code repository - 200 + coding lessons

Real-World Impact:

Netflix: Engineers filter video streaming logs by region, device type, and error category simultaneously
Uber: Operations teams slice ride logs by city, driver status, and trip duration ranges
Slack: Support teams filter message logs by workspace size, feature usage, and performance metrics

The magic happens when you can combine filters dynamically. Instead of writing complex queries, you click "Error Level: Critical" + "Service: Payment" + "Last 1 Hour" and instantly see exactly what you need.

Core Concept: Faceted Search Architecture

Faceted search transforms structured log data into searchable dimensions called "facets." Each facet represents a filterable attribute with pre-computed value counts.

The Three Pillars of Faceted Search

1. Facet Extractor Analyzes incoming logs to identify searchable dimensions:

# Example log entry creates these facets:
{
  "service": "payment-api",
  "level": "error", 
  "region": "us-west-2",
  "response_time": "250ms"
}

# Generates facets:
# service: [payment-api, user-api, order-api]
# level: [error, warn, info, debug]  
# region: [us-west-2, us-east-1, eu-west-1]
# response_time: [0-100ms, 100-500ms, 500ms+]

2. Facet Index Manager Maintains real-time counts for each facet value combination:

Service facets: payment-api (1,247), user-api (892), order-api (445)
Level facets: error (156), warn (234), info (2,198)
Combined facets: payment-api + error (89), user-api + warn (67)

3. Search Coordinator Orchestrates complex multi-facet queries across distributed indexes while maintaining response speed under 100ms.

Architecture Integration

Your faceted search system integrates into the distributed log processing pipeline as an interactive query layer:

Data Flow:

Log Ingestion → Structured logs enter the system
Facet Extraction → Automated metadata analysis identifies filterable dimensions
Index Updates → Real-time facet counting and aggregation
Search Interface → Interactive filters for complex query building
Result Rendering → Filtered log results with highlighted matches

This sits between your Day 54 query parser (which handles the actual search execution) and Day 56's real-time indexing (which will make facet updates instantaneous).

Implementation Highlights

Backend Architecture

Facet Engine Core : day55-faceted-search/backend/app/services/facet_engine.py

class FacetEngine:
    def __init__(self, redis_url: str = "redis://localhost:6379", db_path: str = "data/logs.db"):
        self.redis_client = redis.from_url(redis_url, decode_responses=True)
        self.db_path = db_path
        self.facet_definitions = {
            'service': {'type': 'categorical', 'display': 'Service'},
            'level': {'type': 'categorical', 'display': 'Log Level'},
            'region': {'type': 'categorical', 'display': 'Region'},
            'response_time_range': {'type': 'numeric', 'display': 'Response Time'},
            'hour_of_day': {'type': 'temporal', 'display': 'Time of Day'}
        }
        self.init_database()
        
    def init_database(self):
        """Initialize SQLite database for log storage"""
        conn = sqlite3.connect(self.db_path)
        conn.execute('''
            CREATE TABLE IF NOT EXISTS logs (
                id TEXT PRIMARY KEY,
                timestamp TEXT,
                service TEXT,
                level TEXT,
                message TEXT,
                metadata TEXT,
                source_ip TEXT,
                request_id TEXT,
                region TEXT,
                response_time INTEGER
            )
        ''')
        conn.execute('CREATE INDEX IF NOT EXISTS idx_service ON logs(service)')
        conn.execute('CREATE INDEX IF NOT EXISTS idx_level ON logs(level)')
        conn.execute('CREATE INDEX IF NOT EXISTS idx_timestamp ON logs(timestamp)')
        conn.commit()
        conn.close()

Search Service Integration

class SearchService:
    def __init__(self, facet_engine: FacetEngine):
        self.facet_engine = facet_engine
        
    async def search(self, search_request: SearchRequest) -> SearchResponse:
        """Perform faceted search on logs"""
        start_time = time.time()
        
        # Get filtered logs
        logs = await self.facet_engine.search_logs(search_request)
        
        # Get facets with current filters applied
        facets_summary = await self.facet_engine.get_facets(search_request.filters)
        
        query_time = (time.time() - start_time) * 1000  # Convert to milliseconds
        
        return SearchResponse(
            logs=logs,
            total_count=facets_summary.total_logs,
            facets=[facet.dict() for facet in facets_summary.facets],
            query_time_ms=round(query_time, 2),
            applied_filters=search_request.filters
        )

Frontend React Components

Interactive Facet Panel :

day55-faceted-search/frontend/src/components/FacetPanel.js

import React from 'react';

const FacetPanel = ({ facets, selectedFilters, onFilterChange }) => {
  const handleFacetValueToggle = (facetName, value) => {
    const currentValues = selectedFilters[facetName] || [];
    const newValues = currentValues.includes(value)
      ? currentValues.filter(v => v !== value)
      : [...currentValues, value];
    
    onFilterChange(facetName, newValues);
  };

  if (!facets || facets.length === 0) {
    return <div className="loading">Loading filters...</div>;
  }

  return (
    <div className="facet-panel">
      {facets.map((facet) => (
        <div key={facet.name} className="facet-group">
          <div className="facet-title">{facet.display_name}</div>
          {facet.values.slice(0, 10).map((facetValue) => (
            <div
              key={facetValue.value}
              className={`facet-value ${facetValue.selected ? 'selected' : ''}`}
              onClick={() => handleFacetValueToggle(facet.name, facetValue.value)}
            >
              <input
                type="checkbox"
                className="facet-checkbox"
                checked={facetValue.selected}
                onChange={() => {}} // Handled by onClick above
              />
              <span className="facet-label">{facetValue.value}</span>
              <span className="facet-count">{facetValue.count}</span>
            </div>
          ))}
          {facet.values.length > 10 && (
            <div className="facet-more">
              +{facet.values.length - 10} more
            </div>
          )}
        </div>
      ))}
    </div>
  );
};

export default FacetPanel;

Performance Optimizations

Memory Management

Facet Caching: Keep frequently accessed facet counts in Redis for sub-millisecond access
Index Strategy: Create composite indexes on commonly filtered combinations (service + level, region + timestamp)
Query Optimization: Combine log retrieval and facet counting in single database round-trip

Response Time Targets

Initial Search: < 100ms response time
Filter Updates: < 50ms for facet recalculation
Large Datasets: Maintain performance up to 1M+ logs

Horizontal Scaling Results

1 Consumer:  32 msg/sec
2 Consumers: 65 msg/sec  
4 Consumers: 127 msg/sec
8 Consumers: 248 msg/sec

Scaling efficiency: 95%+ linear improvement

Github Link :

https://github.com/sysdr/course-p/tree/main/day55/day55-faceted-search

Source Code Repository :

git clone https://github.com/sysdr/course.git 

checkout day55  

cd day55/day55-faceted-search 

./build.sh
./run.sh
.demo.sh

🚀 Build, Test & Demo Guide

Quick Setup Commands

# Project initialization
mkdir day55-faceted-search && cd day55-faceted-search
mkdir -p {backend/{app,tests},frontend/{src,public},tests/{unit,integration},docker,data}

# Backend dependencies (Python 3.11+)
pip install fastapi==0.104.1 uvicorn==0.24.0 redis==5.0.1 \
    pydantic==2.5.0 pytest==7.4.3 structlog==23.2.0

# Frontend dependencies (Node.js 18+)
npm install react@18.2.0 react-dom@18.2.0 axios@1.6.0 \
    @mui/material@5.14.15 recharts@2.8.0

Core Implementation Structure

day55-faceted-search/
├── backend/
│   ├── app/
│   │   ├── api/           # FastAPI routes
│   │   ├── models/        # Pydantic models
│   │   ├── services/      # Facet engine & search
│   │   └── main.py        # Application entry
│   └── tests/             # Backend tests
├── frontend/
│   ├── src/
│   │   ├── components/    # React components
│   │   ├── services/      # API integration
│   │   
│   └── public/            # Static assets
└── docker/                # Container configuration

Step-by-Step Build Process

1. Backend Services Setup

# Start Redis for facet caching
redis-server --daemonize yes

# Initialize SQLite database with indexes
python backend/app/services/facet_engine.py init-db

# Start FastAPI backend
cd backend && python -m app.main

Expected Output:

INFO:     Uvicorn running on http://0.0.0.0:8000
INFO:     Application startup complete
✅ Facet engine initialized with 5 facet types
✅ SQLite database ready with optimized indexes

2. Frontend Development Server

# Start React development server  
cd frontend && npm start

Expected Output:

Compiled successfully!
Local:            http://localhost:3000
On Your Network:  http://192.168.1.x:3000
✅ Frontend dashboard accessible

Testing & Verification

Backend API Testing

# Generate sample data
curl -X POST "http://localhost:8000/api/logs/generate?count=500"

# Test faceted search
curl -X POST "http://localhost:8000/api/search/" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "", 
    "filters": {"level": ["error"], "service": ["payment-api"]}, 
    "limit": 10
  }'

Performance Load Testing

# Test concurrent search performance
python tests/load/test_concurrent_search.py

# Expected metrics:
# ✅ 95th percentile: < 100ms
# ✅ Concurrent users: 100+
# ✅ Memory usage: < 200MB
# ✅ Throughput: 65+ queries/second

Integration Testing

# Run comprehensive test suite
python -m pytest tests/ -v --cov=app --cov-report=html

# Expected results:
# ======================== 12 passed in 2.45s ========================
# Coverage: 92%

🎬 Live Demo Experience

Interactive Dashboard Features

1. Real-Time Facet Filtering

Open http://localhost:3000
Click "Generate Sample Data (500 logs)" button
Use facet panels on left to filter by:
- Service: payment-api, user-api, order-api
- Level: error, warn, info, debug
- Region: us-west-2, us-east-1, eu-west-1
- Response Time: 0-100ms, 100-500ms, 500ms+

2. Multi-Dimensional Search

Select multiple filters simultaneously
Watch result counts update in real-time
Observe facet counts recalculate based on current filters
Use search box for text filtering combined with facets

3. Performance Monitoring

Stats bar shows: Total Logs, Query Time, Active Filters
Response times consistently under 100ms
Smooth interactions with instant visual feedback

Google Cloud Skills Boost UI Styling

The interface features:

Clean Material Design with subtle shadows and rounded corners
Responsive Grid Layout adapting to desktop and mobile
Professional Color Scheme using Google's blue (#1a73e8) primary palette
Interactive Elements with hover states and smooth transitions
Accessibility Features with proper contrast and keyboard navigation

Production Deployment

Docker Container Setup

# Build and deploy with Docker Compose
docker-compose up --build -d

# Verify all services running
docker-compose ps

Container Status:

NAME                    STATUS      PORTS
redis-1                Up          0.0.0.0:6379->6379/tcp
faceted-backend-1      Up          0.0.0.0:8000->8000/tcp  
faceted-frontend-1     Up          0.0.0.0:3000->3000/tcp

Health Verification

# Backend health check
curl http://localhost:8000/health
# Response: {"status": "healthy", "service": "faceted-search"}

# Frontend accessibility  
curl -I http://localhost:3000
# Response: HTTP/1.1 200 OK

# Search system stats
curl http://localhost:8000/api/search/stats | jq

🎯 Assignment: E-Commerce Faceted Search

Mission: Build a faceted search system for an e-commerce platform's log analysis.

Requirements:

Extract facets from order processing logs (service, status, region, amount_range)
Create interactive filter interface with real-time counting
Support combined filters (service=payment AND status=failed AND region=us-west)
Demonstrate 1000+ logs/second facet update performance
Show drill-down patterns (region → country → city)

Success Metrics:

Filter combinations return results in <100ms
Facet counts update within 5 seconds of new logs
Interface handles 10+ simultaneous facet filters
Zero query errors under sustained load

Solution Approach:

Service identification: user, inventory, payment
Routing keys: payment.processor.error, inventory.stock.update
Topic patterns: payment.*, inventory.*, user.activity.*
Specialized processing: payment errors → security, inventory → analytics
Performance testing: sustained 1000 msg/sec with <10ms routing latency

Key Takeaways & Next Steps

What You've Accomplished Today

You've built a production-ready faceted search system that demonstrates patterns used by major tech companies:

✅ Multi-dimensional filtering across categorical, numeric, and temporal facets
✅ Real-time facet counting with Redis caching for sub-50ms updates

✅ Interactive React interface with Google Cloud Skills Boost styling

✅ Performance optimization achieving 100+ concurrent user support

✅ Horizontal scaling with 95%+ linear improvement efficiency

Integration Context

Yesterday's Foundation: SQL-like query parser provides the execution engine Today's Enhancement: Visual, multi-dimensional filtering interface
Tomorrow's Evolution: Real-time facet updates as logs arrive

Real-World Applications

The faceted search patterns you implemented today power operational dashboards at companies processing billions of events daily:

GitHub: Routes deployment logs to CI/CD dashboards while sending performance metrics to capacity planning
Airbnb: Separates user activity logs from infrastructure monitoring for targeted team access
Shopify: Filters order processing logs by merchant size, geography, and payment method for fraud detection

Understanding faceted search architecture prepares you for senior roles where you'll design systems making complex data accessible to entire organizations.

Next: Day 56 - Implement real-time indexing of incoming logs

Hands On System Design - Distributed Systems Implementation

Day 55: Building Multi-Dimensional Search …