What We’re Building Today
Yesterday you built distributed semaphores that cap how many workers can use a resource at once. Today we tackle a different question: how long should any one worker be allowed to hold a resource before the system takes it back automatically? We’re building a Lease Manager — a service that grants time-bounded, exclusive ownership over a resource, and reclaims it the moment the clock runs out, crash or no crash.
By the end of today you’ll have a FastAPI service on port 8184, backed by SQLite, that can grant, renew, release, and auto-expire leases, plus a live dashboard showing countdown bars ticking down in real time.
Core Concept: What Is a Lease?
A lock says “you have this until you let go.” A lease says “you have this until the clock says otherwise — whether you let go or not.” That single difference solves one of the nastiest problems in distributed systems: what happens when the resource holder crashes, freezes, or gets network-partitioned away mid-task, and never calls unlock()?
With a plain lock, the resource stays stuck forever. With a lease, it self-expires. This is exactly how Google’s Chubby lock service, Kubernetes’ leader-election leases, and etcd’s session mechanism all work — none of them trust a client to clean up after itself.
The Insight That Matters: Fencing Tokens
Leases introduce a subtle danger: a holder can be paused (garbage collection pause, slow disk, frozen VM) for longer than its lease TTL. The lease manager reasonably assumes it’s dead and re-grants the resource to someone else. Then the original holder “wakes up” and keeps writing — now there are two writers touching the same resource.
The fix is a fencing token: a monotonically increasing number handed out with every grant and renewal. Downstream systems (a database, a file, a piece of hardware) are taught a simple rule: reject any write that carries a token lower than the highest one they’ve already seen. The lease manager doesn’t need to trust that a paused process stays paused — it only needs the resource itself to enforce ordering.
Where This Fits in Your Log Processing System
Every distributed log pipeline has moments where exactly one component should temporarily own something: a shard leader that owns writes to a partition, a compaction worker that owns a segment file while merging it, a scheduler node that owns a batch job. Leases are the primitive that makes “own this for a while, then let someone else try” safe by default.
Concretely, this builds directly on your Week 27 work: distributed semaphores (Day 183) limited concurrent access; leases add the missing time dimension, so tomorrow’s service discovery (Day 185) can safely register instances that automatically disappear from the registry if they stop renewing their lease — no manual deregistration required.
