What We Build Today
A log transport layer that compresses, batches, and prioritises log payloads before sending
Delta encoding so repeated log fields transmit only their changes
An adaptive scheduler that reads available bandwidth and adjusts batch size and interval in real time
A receiver endpoint that decompresses, reconstructs, and stores logs with delivery acknowledgement
A dashboard showing transport efficiency metrics: bytes saved, compression ratio, queue depth
The Constrained-Network Reality
Yesterday you built a collector that buffers logs when the network disappears. Today the network is back, but it is slow, metered, or shared with operational traffic. Sending raw JSON logs over a 50 kbps GPRS link is not just inefficient; it actively competes with the control messages your devices depend on.
Stripe’s global payment terminals face exactly this. Many operate over cellular data with carrier-imposed byte caps. Their log shipping layer reduces payload size by 70 to 80 percent using field-level delta encoding combined with per-batch zlib compression. The insight is that consecutive logs from the same device share most of their structure: the same service name, the same host, the same log level prefix. Only timestamps, counters, and occasional message text change between entries.
Core Concepts
Delta Encoding for Structured Logs
Rather than transmitting each log as an independent JSON object, you record a reference frame and transmit only the fields that differ from it. A hundred consecutive INFO logs from payment-service on device-42 may share seven of nine fields. You send those seven fields once, then only two values per subsequent log.
Adaptive Batching
Fixed batch intervals waste bandwidth when logs are sparse and cause latency spikes when logs are dense. An adaptive scheduler measures the rolling average log rate and adjusts the flush interval so each batch targets a configurable payload budget, say 4 KB compressed. When the network is fast, smaller intervals keep latency low. When the link degrades, intervals widen automatically.
Priority Lanes
Not all logs justify bandwidth. ERROR and CRITICAL logs ship immediately on a high-priority lane with no batching. INFO and DEBUG logs enter the batching pipeline and may be held for seconds or shed entirely when queue depth exceeds a threshold. This mirrors how Cloudflare’s edge nodes handle telemetry: security events bypass the batch queue entirely and stream to the ingestion cluster as they occur.
Compression Strategy
zlib level 6 is the practical sweet spot for IoT: it achieves 60 to 75 percent compression on JSON log streams at roughly 1 MB/s throughput on a Raspberry Pi-class CPU. LZ4 is faster but compresses 15 to 20 percent less. For payloads under 200 bytes, skip compression entirely since the zlib header overhead cancels the gain.
