resilience
What every service needs to stay up under load and failure, in pure
Louise with log as its only dependency (ADR 0081). Promoted from the
identical copies in starfleet, toki2 and the newsletter.
import resilience/gate
import resilience/rate_limit.{Admitted, Wait}
import resilience/retry
// At most 4 argon2 hashes at once: a burst of logins is refused (503)
// rather than piling up.
let hashing = gate.start 4
match gate.admitted hashing fn { argon2.hash password }
{ Ok(hash) -> ..., Error(gate.Full) -> ... }
// A token bucket per key, refilled continuously.
let limit = rate_limit.per_minute 10
let (limit, answer) = rate_limit.taken limit client now_ms
match answer { Admitted -> ..., Wait(seconds) -> ... }
// A start-up step that may fail for a while (the database coming up).
use pool <- retry.retried "database" [500, 1000, 2000, 4000] fn { connect () }
resilience/gate: at mostlimitjobs at once, in an actor; a job that finds every slot taken is refused (Full) at once. A slot frees itself afterlifetime_ms(start_with; 30 s withstart), even when its job's process died.resilience/rate_limit: token buckets per key, a minute's worth at most;takenanswersAdmittedorWait(seconds),wait_forhow long a bucket's units take,costa request's units.resilience/retry: an action tried until it succeeds, waiting each of a list of delays between tries, each failure logged atwarn; the last failure is returned.
Buckets shared by every instance, in PostgreSQL, are packages/throttle.