Day 14: Building Your First Load Generator - Stress Testing Your Distributed System

Week 2 Finale : Performance Benchmark System

After a week of building network-based log collection components, today we'll create a load generator to stress-test our distributed system and measure its true performance limits.

Why Load Testing Matters in Production Systems

Think of your system like a highway. You wouldn't open a new highway without testing how many cars it can handle during rush hour. Companies like Netflix test their systems by intentionally breaking them (Chaos Engineering) to understand failure points before customers experience outages.

Load testing reveals:

• Bottlenecks: Where your system chokes under pressure

• Capacity limits: Maximum throughput before degradation

• Resource utilization: CPU, memory, and network constraints

• Failure modes: How gracefully your system degrades

Core System Design Concepts

Throughput measures how much work your system completes per unit of time - logs processed per second in our case. Latency measures how long individual requests take. These metrics often trade off against each other: higher throughput sometimes means higher latency.

A load generator simulates real-world traffic patterns. Unlike simple benchmarks that measure theoretical maximums, load generators test realistic scenarios with varying loads, network conditions, and failure modes.

System Architecture Overview

Our complete system now includes:

Load Generator → Log Shipper → TCP/UDP Server → Storage
     ↓              ↓              ↓            ↓
   Metrics    Compression     TLS Encryption  Persistence

Project Setup Commands

# Create project structure
mkdir -p distributed-logs-week2/{src,tests,docker,certs,logs,config}
cd distributed-logs-week2

# Create source files
touch src/{load_generator.py,log_shipper.py,tcp_server.py,udp_server.py,benchmark.py}
touch tests/{test_load_generator.py,test_integration.py}
touch docker/{Dockerfile.generator,Dockerfile.server,docker-compose.yml}
touch config/{server.conf,client.conf}
touch requirements.txt README.md

# Generate TLS certificates for testing
mkdir -p certs
openssl req -x509 -newkey rsa:4096 -keyout certs/server.key -out certs/server.crt -days 365 -nodes -subj "/CN=localhost"

requirements.txt

asyncio
aiofiles
psutil
numpy
matplotlib
pytest
pytest-asyncio

TCP Server with TLS and Compression