Master the architecture required to sustain 100 million requests per second. Dive deep into the Go runtime and Linux kernel to eliminate hidden latencies. Start building systems that push the absolute boundaries of modern hardware. Hands-On Distributed Systems Engineering in Go - https://systemdrd.com/courses/system-enginnering-go
What We’re Building Today
Imagine you’re a security analyst at a financial services company. At 3 AM, your system detects a credential-stuffing attack hitting login endpoints from 50,000 different IP addresses. Manual response would take 20 minutes—by then, thousands of accounts could be compromised. Today, we’re building the automated response system that reacts in milliseconds, executing standardized playbooks that isolate threats, alert teams, and preserve evidence—all before a human even wakes up.
Your Mission: Build an automated incident response orchestrator that detects security events from your IOC scanner (Day 159) and executes pre-defined playbooks—blocking malicious IPs, isolating compromised accounts, triggering alerts, and creating forensic snapshots.
The 3 AM Problem
When Cloudflare detected the Okta breach attempt in their systems, their automated response playbooks executed within 34 seconds—blocking suspicious sessions, rotating credentials, and alerting the security team. Manual response would have taken 15-20 minutes. That speed difference prevented a major breach.
Automated incident response transforms security from reactive firefighting into proactive defense. Instead of security analysts manually executing 20-step procedures at 3 AM, playbooks encode expert knowledge into executable workflows that run consistently every time.
Core Architecture
Our incident response system operates as the action layer above your IOC detection system. When IOC scanning identifies threats, it triggers our response coordinator, which selects appropriate playbooks and orchestrates multi-step responses.
Key Components
Playbook Engine: Stores and executes response procedures as code. Each playbook defines triggers (what security events activate it), conditions (when to execute specific actions), and actions (what to do). Think of playbooks as recipes—standardized procedures that produce consistent results.
Response Coordinator: Receives security events, evaluates which playbooks apply, checks prerequisites, and orchestrates execution. It manages the state machine that transitions incidents from detection through analysis, containment, and recovery.
Action Executors: Implement specific response actions—network isolation, account suspension, IP blocking, alert generation, evidence collection. Each executor knows how to safely perform one action and report results.
Audit Logger: Records every action taken during incident response, creating an immutable chain of custody for forensic analysis and compliance requirements.
Playbook Workflow
The Response Lifecycle
When your IOC scanner detects a threat signature, the event flows to the Response Coordinator:
1. Event Ingestion: Security event arrives with IOC details (malicious IP, compromised credential, suspicious pattern). The coordinator parses context—severity level, affected systems, threat type.
2. Playbook Selection: Based on event attributes, the coordinator identifies applicable playbooks. For example, a brute-force attack triggers the “Account Protection Playbook,” while a data exfiltration attempt triggers “Network Isolation Playbook.”
3. Condition Evaluation: Before executing, playbooks check prerequisites. Is the affected user a critical service account? Is this during business hours? Has this IP been seen before? These conditions prevent automated responses from causing unnecessary disruption.
4. Action Execution: The coordinator runs playbook actions sequentially or in parallel. Each action returns success/failure status, enabling the playbook to adapt—if IP blocking fails, escalate to network-wide containment.
5. Human Escalation: Critical incidents or failed automated responses trigger alerts to security teams via PagerDuty, Slack, or email, providing context and recommended next steps.
6. Evidence Preservation: Throughout the response, the system captures logs, network traffic, system snapshots—creating forensic packages for post-incident analysis.
Real-World Response Patterns
Production incident response systems implement these battle-tested playbooks:
Brute Force Attack Response
Detect repeated login failures from single IP
Temporarily rate-limit the source
Alert user if legitimate account targeted
Block IP after threshold violations
Create forensic snapshot of attempts
Data Exfiltration Response
Detect unusual outbound data transfer
Capture network traffic samples
Isolate affected system from network
Suspend involved user accounts
Alert security operations center
Preserve evidence for investigation
Malware Detection Response
Identify process signatures matching IOCs
Terminate malicious processes
Quarantine affected files
Disable network access for infected host
Trigger endpoint detection response (EDR) tools
Schedule malware analysis
Integration with IOC Scanning
Yesterday’s IOC scanner detects threats; today’s response system neutralizes them. The integration creates a continuous security loop:
IOC Scanner → Threat Detection → Response Coordinator → Playbook Execution → Action Logging → Compliance Reports
When your IOC scanner identifies a known C2 (command and control) server IP in outbound connections, it publishes a high-severity event. The response coordinator immediately executes the “C2 Communication Playbook”:
Blocks the C2 IP at firewall level
Identifies all internal hosts communicating with it
Isolates those hosts from the network
Alerts security team with affected host list
Creates packet captures for analysis
Implementation Strategy
We’ll build a Python-based orchestration engine with React dashboard for real-time monitoring:
Backend (Python 3.11):
PlaybookEngine class managing playbook definitions
ResponseCoordinator handling event processing
Action executors for common responses (block IP, isolate host, alert)
Integration with IOC scanner event stream
RESTful API for playbook management
Frontend (React):
Real-time incident dashboard showing active responses
Playbook editor for security teams
Response history with forensic timelines
Manual override controls for critical situations
Action Types We’ll Implement:
Network actions (block IP, isolate subnet, disable interface)
Identity actions (suspend account, reset credentials, require MFA)
Alert actions (email, Slack, PagerDuty, SMS)
Evidence actions (snapshot system, capture traffic, preserve logs)
Testing Incident Response
Professional security teams test playbooks like pilots practice emergency procedures. We’ll implement:
Tabletop Exercises: Simulated security events that trigger playbooks without executing destructive actions—verifying logic without risk.
Canary Systems: Test environments that mirror production, where playbooks execute fully to validate effectiveness.
Automated Verification: After each playbook execution, verify expected state changes (IP actually blocked, account actually suspended) and rollback if verification fails.
Production Considerations
Real-world incident response systems balance automation with safety:
Rate Limiting: Prevent playbooks from blocking entire customer bases during false positives. Implement circuit breakers that pause automated responses if action volume exceeds thresholds.
Approval Workflows: Critical actions (shutting down production services, blocking major IP ranges) require human approval even in automated playbooks.
Rollback Capabilities: Every automated action should have a corresponding undo operation. If blocking an IP causes legitimate business disruption, security teams can quickly reverse it.
Compliance Logging: Every action must be logged with timestamp, triggering event, executing playbook, and result—meeting audit requirements for SOC2, PCI-DSS, and HIPAA.
Hands-On Implementation
GitHub Link:
https://github.com/sysdr/course/tree/main/day160/day160_incident_responsePrerequisites
Before starting, ensure you have:
Python 3.11 or higher installed
Basic understanding of async programming
4GB RAM minimum for running the complete system
Text editor or IDE (VS Code recommended)
Project Structure
Create your workspace:
mkdir day160_incident_response && cd day160_incident_response
mkdir -p src/{playbooks/{templates,engine},actions/{network,identity,alert,evidence},api,dashboard/static}
mkdir -p tests config logs
Your directory structure should look like:
day160_incident_response/
├── src/
│ ├── playbooks/
│ │ ├── engine/
│ │ │ └── playbook_engine.py
│ │ ├── templates/
│ │ │ ├── brute_force_response.yaml
│ │ │ ├── malware_response.yaml
│ │ │ └── c2_communication_response.yaml
│ │ └── response_coordinator.py
│ ├── actions/
│ │ ├── network/
│ │ ├── identity/
│ │ ├── alert/
│ │ └── evidence/
│ ├── api/
│ │ └── main.py
│ └── dashboard/
│ └── static/
│ └── index.html
├── tests/
├── config/
└── logs/
Dependencies Installation
Create requirements.txt:
fastapi==0.111.0
uvicorn[standard]==0.30.1
pydantic==2.7.4
aiohttp==3.9.5
redis==5.0.6
pyyaml==6.0.1
pytest==8.2.2
pytest-asyncio==0.23.7
structlog==24.2.0
Install dependencies:
python3.11 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
Core Component: Playbook Engine
The playbook engine is the brain of our system. Create src/playbooks/engine/playbook_engine.py:
Key features:
YAML-based playbook templates
Conditional action execution
Automatic retry with exponential backoff
Complete audit logging
The engine loads playbook templates, substitutes event context into parameters, and orchestrates action execution with proper error handling.
Response Coordinator
Create src/playbooks/response_coordinator.py:
The coordinator acts as the event router. When security events arrive, it:
Queues events for processing
Matches events to appropriate playbook templates
Initiates parallel playbook execution
Tracks active responses
Maintains response history
Action Executors
Network Actions (src/actions/network/network_actions.py)
Implements:
BlockIPAction: Adds firewall rules to block malicious IPsIsolateSystemAction: Disconnects compromised systems from networkCaptureTrafficAction: Records network traffic for forensics
Identity Actions (src/actions/identity/identity_actions.py)
Implements:
SuspendAccountAction: Disables compromised user accountsForcePasswordResetAction: Requires password changesRevokeSessionsAction: Terminates active user sessions
Alert Actions (src/actions/alert/alert_actions.py)
Implements:
SendEmailAlertAction: Notifies security team via emailSendSlackAlertAction: Posts alerts to Slack channelsCreatePagerDutyIncidentAction: Creates high-priority incidents
Evidence Actions (src/actions/evidence/evidence_actions.py)
Implements:
CreateSystemSnapshotAction: Captures full system statePreserveLogsAction: Archives relevant log entriesCollectArtifactsAction: Gathers forensic evidence
Playbook Templates
Create YAML templates in src/playbooks/templates/. Each template defines:
Trigger conditions (what events activate it)
Actions to execute (in order)
Conditional logic (when to run specific actions)
Timeout and retry configuration
Example brute force response template (brute_force_response.yaml):
name: brute_force_response
description: Response to brute force authentication attacks
severity: high
triggers:
- brute_force_attack
actions:
- name: block_attacker_ip
type: block_ip
parameters:
ip_address: ${source_ip}
duration: 7200
timeout: 10
max_retries: 2
- name: alert_security_team
type: send_email_alert
parameters:
recipients: ['security@company.com']
subject: 'Brute Force Attack Detected'
priority: high
timeout: 10
FastAPI Application
Create src/api/main.py:
The API provides:
POST /api/events- Submit security eventsPOST /api/playbooks/execute- Manually trigger playbooksGET /api/responses- List incident responsesGET /api/audit-log- Retrieve audit trailGET /api/metrics- System performance metrics
Real-Time Dashboard
Create src/dashboard/static/index.html:
The dashboard displays:
Active incident responses
Playbook execution status
Test scenario buttons
Recent audit log entries
System metrics
Built with React (loaded via CDN) for real-time updates without page refreshes.
Build and Test
Running Tests
# Set Python path
export PYTHONPATH="$(pwd):$PYTHONPATH"
# Run test suite
python -m pytest tests/ -v
# Expected output:
# tests/test_playbook_engine.py::test_playbook_execution PASSED
# tests/test_playbook_engine.py::test_action_retry PASSED
# tests/test_playbook_engine.py::test_condition_evaluation PASSED
Starting the System
# Start the API server
python -m uvicorn src.api.main:app --host 0.0.0.0 --port 8000 --reload
Expected output:
INFO: Uvicorn running on http://0.0.0.0:8000
INFO: Application startup complete.
Accessing the Dashboard
Open your browser to:
http://localhost:8000
You should see:
Metrics showing 0 active playbooks (initially)
Three test scenario buttons
Empty response history (until you trigger incidents)
Functional Testing
Test Scenario 1: Brute Force Attack
Click “Test Brute Force Attack” in the dashboard, or use curl:
curl -X POST http://localhost:8000/api/events \
-H "Content-Type: application/json" \
-d '{
"event_type": "brute_force_attack",
"severity": "high",
"source": "192.168.1.100",
"details": {
"source_ip": "192.168.1.100",
"target_user": "admin",
"failed_attempts": 15
}
}'
Watch the dashboard update in real-time showing:
Playbook execution status
Individual action results
Audit log entries
Test Scenario 2: Malware Detection
curl -X POST http://localhost:8000/api/events \
-H "Content-Type: application/json" \
-d '{
"event_type": "malware_detected",
"severity": "critical",
"source": "workstation-42",
"details": {
"system_id": "workstation-42",
"user_id": "jsmith",
"malware_type": "ransomware"
}
}'
Expected actions:
System isolation within 1 second
Traffic capture initiated
Forensic snapshot created
User account suspended
PagerDuty incident created
Test Scenario 3: C2 Communication
curl -X POST http://localhost:8000/api/events \
-H "Content-Type: application/json" \
-d '{
"event_type": "c2_communication",
"severity": "critical",
"source": "server-05",
"details": {
"system_id": "server-05",
"c2_ip": "203.0.113.10",
"user_id": "system"
}
}'
This triggers the most comprehensive response:
C2 IP blocked at firewall
Affected system isolated
Network traffic captured
System snapshot created
Forensic artifacts collected
Security operations alerted
Verification
Check System Metrics
curl http://localhost:8000/api/metrics | python -m json.tool
Expected output:
{
"active_playbooks": 0,
"total_responses": 3,
"audit_entries": 12,
"playbook_templates": 3
}
Review Audit Log
curl http://localhost:8000/api/audit-log | python -m json.tool
Each entry shows:
Timestamp of action
Playbook and action name
Execution status (success/failed)
Parameters used
Result details
Inspect Response Details
curl http://localhost:8000/api/responses | python -m json.tool
Shows complete response history including:
Event details that triggered response
Playbooks executed
Action results
Execution duration
Docker Deployment (Optional)
For production-like deployment:
Create Dockerfile:
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
EXPOSE 8000
CMD ["python", "-m", "uvicorn", "src.api.main:app", "--host", "0.0.0.0", "--port", "8000"]
Create docker-compose.yml:
version: '3.8'
services:
incident-response:
build: .
ports:
- "8000:8000"
volumes:
- ./logs:/app/logs
Run with Docker:
docker-compose up --build
Performance Verification
Your system should achieve:
Response initiation: < 100ms after event detection
Playbook execution: < 5 seconds for standard playbooks
Action execution: < 2 seconds per action
Audit logging: 100% of all actions
Success rate: > 95% for standard actions
Working Demo Link :
What You’ve Accomplished
By lesson’s end, you’ll have:
Automated incident response orchestrator processing security events
Library of executable security playbooks (brute force, data exfiltration, malware)
Real-time dashboard monitoring active incident responses
Integration with Day 159’s IOC scanner
Complete audit trail of all automated actions
Foundation for Day 161’s compliance reporting
Performance Target: Respond to detected threats within 5 seconds, execute multi-step playbooks in under 30 seconds, maintain complete audit logs for compliance.
Assignment: Build the Ransomware Response Playbook
Challenge: Create a playbook that responds to ransomware detection (identified by IOC scanner finding known ransomware signatures).
Required Steps:
Immediately isolate affected system from network
Suspend user account associated with infected system
Create forensic snapshot of system state
Block command-and-control IPs extracted from malware
Alert security operations center with high priority
Generate incident report with timeline
Success Criteria:
Playbook executes all steps in under 60 seconds
Each action logs execution details for audit
Failed actions trigger appropriate escalation
Dashboard displays real-time progress
Solution Hints:
Define playbook as YAML with actions, conditions, and metadata
Each action should be idempotent (safe to retry)
Use the action executor framework for network, identity, and alert actions
Test with simulated ransomware detection events
Implement verification steps after each action
Create rollback procedures for testing scenarios
Bonus Challenge: Add conditional logic—if the infected system is a critical server, trigger additional approval workflow before network isolation.
Tomorrow’s Preview
Day 161 builds on today’s audit logging to generate automated compliance reports. You’ll transform raw incident response logs into formatted reports that satisfy security frameworks—proving to auditors that your system detects threats and responds appropriately, all while maintaining detailed documentation.
Next: Day 161 - Security Compliance Reporting: Automated frameworks for PCI, SOC2, and HIPAA