Day 1: Setting Up Your Distributed Systems Environment - Building first component

May 13, 2025

Welcome to the first day of our "254-Day Distributed Log Processing System Implementation" journey! Today, we'll set up the foundation for building your distributed system by configuring your development environment.

What Are Distributed Systems?

Imagine you and your friends are organizing a massive school event. If one person tried to handle everything—decorations, food, music, tickets—they'd be overwhelmed. Instead, you divide tasks among team members who coordinate through group chats. That's essentially a distributed system: multiple computers working together as one cohesive unit, sharing information to solve problems too large for a single machine.

Why Docker, Git, and VS Code?

Before building a skyscraper, you need proper tools and a solid foundation. For our distributed system:

Docker creates consistent environments across different computers (like having identical science lab kits for everyone)
Git tracks changes and enables collaboration (like keeping organized notes that everyone can share)
VS Code provides a powerful editor with extensions for our work (like having a customizable workbench)

Setting Up Your Environment

Let's install and configure each tool step by step:

1. Install Git

Git is our version control system—it tracks changes to your code and helps with collaboration.

Windows:

Download Git from git-scm.com
During installation, accept the default options

Mac:

Open Terminal and type: xcode-select --install
Follow the prompts to install developer tools including Git

Linux:

Open Terminal and type: sudo apt-get update && sudo apt-get install git (Ubuntu/Debian)
For other distributions, use your package manager

Verify installation by opening a terminal/command prompt and typing:

git --version

2. Install Docker

Docker lets us create containerized environments that work the same way on any computer.

Windows/Mac:

Download Docker Desktop from docker.com
Install and follow the setup wizard
Docker Desktop includes Docker Engine, Docker CLI, Docker Compose, and other tools

Linux:

Follow the official Docker installation guide for your distribution

Verify installation:

docker --version
docker-compose --version

3. Install VS Code

Visual Studio Code is our integrated development environment (IDE).

Download from code.visualstudio.com
Install with default settings

4. Install Essential VS Code Extensions

Open VS Code and install these extensions (click the Extensions icon in the sidebar):

Docker
Remote - Containers
Python
GitHub Pull Requests and Issues
YAML

Creating Your Project Repository

Now let's create and structure your project:

Open Terminal/Command Prompt
Create your project directory:

mkdir distributed-log-processor
cd distributed-log-processor

Initialize Git repository:

git init

Create your project structure:

mkdir -p src/services config data docs tests
touch README.md docker-compose.yml .gitignore

Create a basic .gitignore file:

echo "# Python" > .gitignore
echo "__pycache__/" >> .gitignore
echo "*.py[cod]" >> .gitignore
echo "*$py.class" >> .gitignore
echo ".env" >> .gitignore
echo "venv/" >> .gitignore
echo "data/" >> .gitignore

Create your first Docker Compose file:

docker-compose.yml

version: '3'

services:
  # Our simple Python service to test the environment
  logger:
    build:
      context: ./src/services/logger
    volumes:
      - ./src/services/logger:/app
    ports:
      - "8000:8000"
    environment:
      - LOG_LEVEL=INFO

Create your first Python service:

src/services/logger/app.py

import os
import time
from datetime import datetime

def main():
    """Simple logger service to verify our environment is working."""
    print("Starting distributed logger service...")
    log_level = os.environ.get("LOG_LEVEL", "INFO")
    print(f"Log level set to: {log_level}")
    
    while True:
        timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        print(f"[{timestamp}] Logger service is running. This will be part of our distributed system!")
        time.sleep(5)

if __name__ == "__main__":
    main()

src/services/logger/Dockerfile

FROM python:3.9-slim

WORKDIR /app

# Copy just the requirements.txt first to leverage Docker cache
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Run the application
CMD ["python", "app.py"]

src/services/logger/requirements.txt

# No external dependencies for our simple service yet
# We'll add more as we expand our distributed system

README.md

# Distributed Log Processing System

This project is part of the "365-Day Distributed Log Processing System Implementation" series.

## Project Overview 

We're building a distributed system for processing log data at scale. This repository contains all the code and configuration needed to run the system.

## Getting Started

### Prerequisites
- Docker and Docker Compose
- Git
- VS Code (recommended)

### Running the Application
1. Clone this repository
2. Navigate to the project directory
3. Run `docker-compose up`

## Project Structure
- `src/services/`: Contains individual microservices
- `config/`: Configuration files
- `data/`: Data storage (gitignored)
- `docs/`: Documentation
- `tests/`: Test suites

## Day 1 Milestones
- Set up development environment
- Created project structure
- Implemented basic logger service

Architecture Diagram

Data Flow Diagram

Build It Today: Your First Distributed System Component

Now it's time to bring everything together and test your setup! Follow these steps to see your logger service in action:

Open a Terminal/Command Prompt and navigate to your project directory:

cd distributed-log-processor

Build and start your Docker container:

docker-compose up --build

If everything worked correctly, you should see your logger service start and begin printing timestamped messages every 5 seconds. Congratulations! You've successfully:

Set up a complete development environment
Created a structured project repository
Built a simple containerized service

The Distributed Systems Connection

You might be wondering: "How is this a distributed system? It's just one service!"

Great observation! Today we've built the foundation that will allow us to add multiple services that work together. Think of it like building the first LEGO piece of what will become a massive LEGO city.

In the coming days, we'll add more services that communicate with each other to form a true distributed system. Our logger service will eventually connect to other components and process log data from multiple sources simultaneously—just like how different departments in a school need to coordinate during a big event.

Complete Source Code

Looking Ahead

In our next session, we'll:

Add another service that generates log data
Create a way for our services to communicate
Implement basic error handling and recovery

Success Criteria

You've completed Day 1 successfully if:

✅ Docker, Git, and VS Code are installed and working
✅ You've created a project repository with the correct structure
✅ Your logger service runs in a Docker container
✅ You see timestamp messages appearing every 5 seconds

Real-World Connection

The environment you've just set up mirrors what professional developers use to build systems like:

Netflix: Uses containerized services to stream videos to millions of users
Spotify: Processes music streaming data using distributed log analysis
Instagram: Handles billions of photos with distributed processing systems

Each of these platforms started with a foundation similar to what you've built today!

Remember: The most complex distributed systems in the world start with a single service. You've taken your first step toward building something incredible!