What We’re Building Today
Local SQLite log buffer that persists logs during network outages
Connectivity monitor that detects online/offline transitions in real time
Store-and-forward sync engine with priority-based flushing
Central FastAPI receiver with deduplication and sequence reassembly
React dashboard showing edge device sync status and buffer depth
The Problem Nobody Talks About
Most logging tutorials assume a stable network. Edge devices don’t get that luxury.
An Amazon Flex driver’s handheld scanner logs 400 package-scan events before hitting a dead zone between warehouses. A wind turbine sensor records vibration anomalies for 6 hours before the maintenance truck’s cellular modem comes back in range. A hospital medication cart logs 200 dispense events in an elevator with zero signal.
In all three cases, the logs still need to arrive at the central system—complete, ordered, and deduplicated.
This is the store-and-forward problem, and it’s fundamentally different from what Days 31–36 covered with RabbitMQ. There, the broker was always reachable. Here, the broker might not exist from the device’s perspective for hours.
Core Architecture
Three distinct zones interact in this system:
Edge Zone — The device itself. A lightweight Python agent runs continuously, writing incoming log events to a local SQLite buffer regardless of network state. Each log gets a monotonically increasing local sequence number and a device-scoped UUID.
Sync Engine — A background thread watches the connectivity monitor. The moment the network returns, it reads unsynced logs from the buffer ordered by priority tier (ERROR > WARN > INFO > DEBUG), batches them into compressed payloads, and ships them upstream. On acknowledgment, it marks records as synced.
Central Receiver — A FastAPI endpoint accepts batches from any number of edge devices. It runs a deduplication check using the device ID + sequence number composite key, merges out-of-order arrivals using the embedded timestamp, and commits clean records to the central log store.
How Connectivity Awareness Works
The connectivity monitor doesn’t use a ping to 8.8.8.8 — that’s brittle. Instead it attempts a lightweight HTTP HEAD request to the central receiver’s /health endpoint. This confirms both network reachability and that the receiver is operational.
State transitions matter here:
OFFLINE → ONLINE : trigger immediate sync of entire buffer
ONLINE → OFFLINE : switch to local-only write mode, stop sync attempts
ONLINE → ONLINE : periodic sync tick (every 30 seconds)
The edge agent never drops logs in any state. The buffer is the source of truth on the device.
SQLite as the Edge Buffer
Why SQLite and not a file or in-memory queue? Three reasons:
Crash safety — WAL mode gives you atomic writes. If the device powers off mid-write, no corruption.
Priority queries —
SELECT * FROM logs WHERE synced=0 ORDER BY priority DESC, seq ASC LIMIT 100is ten characters of SQL, not 200 lines of custom heap code.Deduplication — The
device_id + sequnique constraint prevents double-writes on the device side.
Schema is deliberately minimal:
CREATE TABLE edge_logs (
seq INTEGER PRIMARY KEY AUTOINCREMENT,
device_id TEXT NOT NULL,
ts REAL NOT NULL,
level TEXT NOT NULL,
payload TEXT NOT NULL,
synced INTEGER DEFAULT 0,
priority INTEGER DEFAULT 3
);Sequence Reassembly at the Receiver
When three devices each send batches simultaneously after coming back online, the receiver sees interleaved sequence numbers. Tesla’s vehicle telemetry team solved this exact pattern: each vehicle is its own ordering namespace. You never sort across devices—only within a device’s own sequence stream.
The receiver maintains a per-device last_received_seq watermark. Gaps in the sequence trigger a 10-second wait window before committing, allowing retransmissions to fill holes. After the window, any still-missing sequences get flagged as “gap detected” in the central log metadata.
What You’ll See Running
Terminal shows the edge agent cycling through states — writing logs locally, detecting network loss, buffering silently, reconnecting, then flushing the backlog in priority order. The React dashboard shows each simulated device as a card: buffer depth, sync percentage, last seen timestamp, and connectivity state badge (green/red).
The central server logs show batch arrivals with deduplication counts — confirming that even if a device sends the same batch twice (network retry), records appear exactly once in the central store.
Integration with the Bigger System
This edge collector slots in directly below Week 25’s executive dashboard. The C-level views built on Day 176 now gain a new data source: edge devices that were previously invisible during connectivity gaps. The sync engine’s synced_at timestamp lets the dashboard distinguish real-time data from backfilled historical data — a distinction that matters enormously in operations analytics.
Day 178 builds on today’s sync engine by adding compression and delta encoding to the batch payloads, reducing the bytes transferred per log by up to 80%.
Github Link:
https://github.com/sysdr/course-p/tree/main/day177/day177_edge_log_collector
Implementation Guide Step-to-Step
Local edge log buffer with store-and-forward sync to a central FastAPI receiver, plus a React (Vite) dashboard served from the API when frontend/dist is built.
There are no API keys in this demo; run it only on trusted networks or behind your own auth/TLS.
Requirements
Python 3.11+ recommended (3.12 works with the pinned deps in
requirements.txt)Node.js 20+ and npm (only if you build or develop the dashboard)
Docker and Docker Compose plugin (optional, for container run)
Install Python dependencies:
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txtBuild the dashboard
cd frontend
npm install
npm run build
cd ..The API serves / from frontend/dist/index.html and bundles under /assets/ after a successful build.
Development server (alternate):
cd frontend && npm run devUses
http://localhost:5173
and calls the API at
http://127.0.0.1:8100
.
Run (local)
From this directory:
./start.shAPI + dashboard:
http://127.0.0.1:8100/
OpenAPI: http://127.0.0.1:8100/docs
Stop:
./stop.shTests
source venv/bin/activate
pytest tests/ -v --tb=shortDocker
cd docker
docker compose up --buildCompose builds from the parent directory (context: .. in docker-compose.yml). Runtime SQLite and logs use the mounted ../logs volume.
Tear down and free unused Docker disk:
./cleanup.shCleanup script
cleanup.sh runs ./stop.sh, docker compose down for this project (when Docker is available), then docker container|network|image|builder prune to drop unused objects. Review flags before running on shared machines; it does not run docker system prune -a (no aggressive removal of all unused images).
Working Code Demo:
Success Criteria
By the end of this build you’ll have:
Edge agent that writes logs to local SQLite with zero data loss during simulated network outages
Sync engine that detects connectivity and flushes the buffer in priority order
Central receiver that deduplicates and merges batches from multiple devices
Dashboard showing per-device buffer depth and sync status in real time
Assignment
Extend the edge agent with a configurable retention policy: logs older than N hours that haven’t synced should be promoted to priority 1 (highest) and, if still unsynced after an additional M hours, written to a local compressed archive file and removed from the SQLite buffer to prevent unbounded disk growth.
Solution hints: Add a created_at column to the schema. Run a background task every 15 minutes that queries WHERE synced=0 AND created_at < (now - N_hours) and updates their priority. A second pass queries WHERE synced=0 AND created_at < (now - N+M_hours), serializes them to a .jsonl.gz archive using Python’s gzip module, then deletes those rows.
Next: Day 178 — Making log transport bandwidth-efficient with delta encoding and adaptive compression for constrained networks