Benchmarks
Programs timed by scripts/bench.sh, built with --release. Each prints a checksum that must equal its //> lines, so an optimisation cannot change what a program computes: CI builds each optimised and unoptimised and checks both (bench_programs_natively_both_ways in louise_codegen_c/tests/native.rs, issue #219).
| Program | Measures |
|---|---|
lists | allocation and freeing: map, filter, reverse, fold (Perceus reuse and drops) |
arrays | a million pushes, a map and a million sets on unique arrays (updated in place), and folds |
dict | a persistent ordered map, through Ord dictionaries |
arith | arithmetic through the Num trait, and plain recursion |
effects | a tail-resumptive fn operation in a hot loop, and a repeated-resumption ctl generator (the pool non-regression control) |
handles | 200,000 fresh one-shot handlers, then 200,000 fresh multi-shot last-use handlers, with one operation and one resume each (issue #329) |
html | rendering a large page: 200,000 rows, joined once (5,000 rows before issue #36, when string.join was quadratic; timings before and after are not comparable) |
messages | a ring of 1,000 actors passing a token 200 times, on all threads |
floats | boxed Float arithmetic in a loop: allocating and freeing small blocks (ADR 0013) |
matmul | a 64×64 matrix product over flat arrays: float arithmetic in the hot loop (ADR 0045) |
parallel | 64 actors computing, on all threads: spreading work over the scheduler's threads |
fanin | 64 actors sending 20,000 messages each to one collector, on all threads: contention on one actor's mailbox (ADR 0028) |
idle | spawning 100,000 actors that wait for a message: spawn cost; ADR 0028 measures the memory per idle actor with it |
selective | 50,000 calls, each waiting on its own reply subject, while 4 actors flood the caller's other subject: wake-ups for messages a receive does not wait for (ADR 0028) |
timers | timeouts: 1,000 client/server pairs calling with a timeout that never expires, and 2,000 actors that sleep or time out 50 times each (virtual time) |
records | orders (a record, 20 or 100 items, a string) through a pipeline of 8 actors: messages moved, copied by the first hop, and promoted to shared (issue #156) |
churn | 64 actors each spawning 2,000 short-lived children one at a time, on all threads: the actor table's write lock and the stack pool (issue #131) |
ssr | a realistic server-rendered page: a layout, components and 500 rows from an in-memory list, rendered once per request, 1,000 requests (the SSR optimization plan's page) |
msgpack | MessagePack round-trips of a representative nested value (an API payload), 20,000 times: the codec's allocation diet |
sorts | five 100,000-element patterns (random, sorted, reverse, low-cardinality, organ-pipe) through std/list's and std/array's sorts (issue #380) |
strings | string.find and string.contains over an 8 MiB corpus, needles of 1, 4, 16 and 64 bytes, absent or at the very end (issue #380) |
To compare a change: scripts/bench.sh --save before.txt on the base, then scripts/bench.sh --compare before.txt with the change. They are not part of scripts/ci.sh: timings on shared runners are too noisy to gate on.
Allocations
LOUISE_STATS=1 makes a program print louise: allocs: N at exit: the heap blocks it ever allocated, summed over threads (docs/runtime-abi.md). It needs the checked runtime — the one debug builds, run --native and test --native link; a release build links the fast one, which prints unknown — so measure with a debug-built program (louise build prog.lse), or a release one with LOUISE_RUNTIME=checked. Allocations per request are N divided by the requests the run served (ssr renders 1,000). The count is exact and the same on every run of a build, so one run is enough.
The PostgreSQL wire codec's decode bench is not here — it needs the package: packages/postgres/bench (run it there with louise run --native).
To check layout stability: scripts/bench-variance.sh builds every benchmark from scratch a few times, with the fixed code layout and without, runs the builds interleaved, and reports how far each configuration's builds drift apart (issue #193; around 2% is the bar). Pin the timed runs with BENCH_PIN (e.g. BENCH_PIN=0) on a busy machine.
Results over time (issue #101)
.toboggan/workflows/bench.yml runs scripts/bench-track.sh on a dedicated, quiet runner labelled louise-bench, by hand (workflow_dispatch) until that runner is registered; then on each push to main too (the workflow's comment says how, and why not before). The script runs scripts/bench.sh and scripts/bench-compile.sh, and counts each benchmark's user-space instructions with perf stat --no-scale on one scheduler thread (skipped without perf). Counted that way, they are the same to a few parts in 10^8 from run to run; on 22 threads, stealing and spinning vary them by 0.3–1.5%. On a hybrid CPU the per-core-type counts are summed, and not scaled: perf's default scaling, meant for multiplexed counters, counted up to 30% too many.
Results stay on the runner, in LOUISE_BENCH_DIR (default ~/louise-bench): one CSV per commit (commit,benchmark,metric,value; metrics ms, compile_ms, instructions), and index, the commits in the order they ran. Each run is compared with the previous one. Regressions beyond BENCH_THRESHOLD_MS (default 10%) or BENCH_THRESHOLD_INSTRUCTIONS (default 1%) are written to <commit>.report and printed, without failing the run. With the secret LOUISE_BENCH_TOKEN (a personal token) and the variable LOUISE_BENCH_ISSUE (an issue number), the workflow also posts them on that issue.
Run time (issue #61)
What changed, and why: ADR 0013 (specialised evidence, inline Int primitives), ADR 0002's note on the allocator's fast paths, ADR 0003's note on calls that never unwind, and ADR 0004 Revision 4 (fewer wake-ups, cheaper lookups).
scripts/bench.sh --compare, BENCH_RUNS=7, all threads, on a shared 22-thread laptop (load average 3–6). "Before" is the v1 chain this work is stacked on (v1-multishot: big integers, multi-shot resumptions, GCC through cc); a third round builds that base with clang, to separate the compiler from the rest of the work:
| Benchmark | Before (ms) | After (ms) | Change | Second round | Base built with clang |
|---|---|---|---|---|---|
arith | 61 | 16 | −73% | 56 → 17, −69% | 45 → 17, −62% |
arrays | 307 | 152 | −50% | 280 → 150, −46% | 255 → 139, −45% |
dict | 186 | 115 | −38% | 183 → 119, −34% | 183 → 115, −37% |
effects | 48 | 25 | −47% | 47 → 24, −48% | 40 → 25, −37% |
floats | 143 | 68 | −52% | 140 → 68, −51% | 134 → 70, −47% |
html | 312 | 295 | −5% | 294 → 290, −1% | 337 → 302, −10% |
lists | 256 | 179 | −30% | 236 → 176, −25% | 232 → 199, −14% |
messages | 420 | 60 | −85% | 414 → 60, −85% | 445 → 58, −86% |
parallel | 559 | 119 | −78% | 425 → 121, −71% | 409 → 143, −65% |
User-space cycles (one scheduler thread, performance cores, best of 5 interleaved) against the same base built with clang: arith −71%, arrays −45%, dict −34%, effects −43%, floats −49%, html −2%, lists −16%, messages −20%, parallel −60%.
Before this work was rebased on the v1 chain, against its own compile-time branch (clang, Int overflow panicking): arith −83% cycles, dict −46%, effects −42%, floats −54%, lists −20%, messages −21%, parallel −72%.
html is bound by copying strings (tl_html_escape, string.join's collection of parts, memcpy) and was not worked on.
Compile time
scripts/bench-compile.sh times louise build of examples/http_demo and of compile/std_tour.lse (a program using most of std), debug and release, each cold (into an empty directory) and warm (the same build again, unchanged). LOUISE=path/to/louise times another compiler binary, to compare versions. louise build --timings (or LOUISE_TIMINGS=1) prints where the time goes: parse, resolve, typecheck, lower, specialise, perceus, emit C, find cc (the compiler and linker, kept in the build directory), natives, C compile, link. louise check --timings prints load (finding packages, reading files), parse, resolve and typecheck.
How louise build keeps compile times down (issue #61):
- clang first. Without
LOUISE_CC, the C compiler is the first ofclang,ccandgccthat supportsmusttail. On the generated C, clang 22 compiles about 3 times faster than GCC 16 at-O2and 2 times faster at-O0 -g, and its code runs 3–28% fewer cycles on these benchmarks. - Kept objects. Objects go to
build/cache/(ADR 0025), one set per profile, with a hash of the compile command and of the contents of every file the compiler read (its-MMDlist). A build whose hash is unchanged only links. Native C dependencies (kind = "c") use the same check, and compile in parallel.run --nativeandtest --nativekeep their builds there too (issue #113). - Skipped builds. When nothing a build read has changed, the next one stops after parsing (a fingerprint in
build/cache/, ADR 0025 Revision 1, issue #119). - A faster linker.
mold, elselld, when one is onPATH;LOUISE_LDforces one (bfdfor GNU ld). Linkingliblouise_rt.atakes about 40 ms instead of 90 (ADR 0025 note, issues #103, #110). - The compiler probe is kept. The build directory remembers which compiler and linker it found (
build/cache/toolchain), so a warm build runs no probe (issue #111). - Release executables without the runtime's debug information. A release link passes
-Wl,--strip-debug: the program's own C has none, and what was dropped is Rust's std, insideliblouise_rt.a. hello goes from 6.6 MB to 2.5 MB, and links 6 ms faster with mold (issue #115). The runtime's line numbers are gone from release backtraces;LOUISE_CFLAGS=-gkeeps everything. - Unused code left out. Links pass
-Wl,--gc-sections: the runtime archive has a section per function, and a program uses a fraction of it. hello's release executable is 0.98 MB instead of 2.48 MB, the demo's 1.40 MB instead of 2.64 MB, and the link takes 47 ms instead of 55 (issue #125). - Line tables only, for debug builds.
-g1instead of-g: backtraces, sanitizer reports and#linelocations keep their.lselines, and debuggers lose the variables, which the generated C namesv123anyway. The C compile ofhttp_demo, cold, takes 64 ms instead of 72 (best of 5; issue #116).LOUISE_CFLAGS=-gbrings the variables back. - One C unit per module, for large debug builds (ADR 0029). Units are compiled in parallel, and each is named after its module, with function names that don't depend on the rest of the program. Editing one module recompiles its unit only:
std_tourafter an edit, 115 → 93 ms. Release builds stay in one unit: calls between units cannot be inlined, and split in 4 units,arithran 30% more cycles,effects9% anddict7%. Split per module with ThinLTO,arithstill ran 2.6% more instructions, so release builds stay in one unit (ADR 0030, issue #118).LOUISE_C_UNITS=1keeps a debug build in one unit.
Targets and results, on a 22-thread laptop shared with other builds (load average 3–5), best of 5, in ms; "before" is the v1 chain this work is stacked on (v1-multishot: GCC through cc, no cache, one unit):
| Build | Target | Before | After | Change |
|---|---|---|---|---|
http_demo debug, cold | ≤ 50% | 1062 | 275 | −74% |
http_demo debug, warm | ≤ 200 ms | 1051 | 138 | −87% |
http_demo release, cold | ≤ 50% | 3367 | 1048 | −69% |
http_demo release, warm | ≤ 200 ms | 3363 | 133 | −96% |
std_tour debug, cold | ≤ 50% | 302 | 143 | −53% (186 ms, −38%, until mold or lld were used, issue #103) |
std_tour debug, warm | ≤ 200 ms | 300 | 129 | −57% |
std_tour release, cold | ≤ 50% | 1335 | 484 | −64% |
std_tour release, warm | ≤ 200 ms | 1289 | 132 | −90% |
Against main at f2b02b7, before the v1 chain, the results were close: http_demo −67% (debug, cold) and −64% (release, cold), warm builds 0.13 s.
What remained of a warm build was linking liblouise_rt.a (about 80 ms with GNU ld) and finding the C compiler (about 15 ms, one probe compile). With mold or lld and the kept probe (issues #103, #110, #111), a warm build takes 62–75 ms, about 40 of them linking: http_demo 138 → 74 ms, std_tour 124 → 62 ms (best of 5, against LOUISE_LD=bfd). The compiler's own phases take about 20 ms for these programs, and grow linearly with the number of functions (250 to 2,000 generated functions; Perceus is the largest, about 0.15 ms per function). They were quadratic in the length of one function: Perceus took 4 s for a function of 1,000 statements, and a function of 2,000 statements overflowed the compiler's stack. Perceus now takes 57 ms for 8,000 statements, and louise runs on a 1 GiB stack.
Each optimisation's target, set after profiling it and before building it, and the result (cycles as above unless stated):
| Optimisation | Target | Result |
|---|---|---|
| Specialised evidence (ADR 0013) | arith −50% | −71% (effects −33%, dict −22%, lists −8%) |
Inline Int primitives | arith −10% more | −40% more (dict −11%, effects −10%, lists −9%) |
| Allocator fast paths | floats −20%, nothing slower | −39%; nothing slower beyond ±1% |
| Receives without table lookups, cheap id hashing | messages −10% on one thread | −22% |
| Links as a set | tests/stress/actors.lse twice as fast on one thread | 2.2–2.8 s → 1.0 s; 10–20% slower on 4–22 threads (ADR 0004 Revision 4), then 7–14% faster on 2–22 threads than that with one link inline (issue #104) |
| Fewer wake-ups | messages on 22 threads within 1.5× of one thread, parallel not slower | 63 ms against 64 ms; parallel 113 → 104 ms |
| Calls that never unwind | 20% fewer yield checks in http_demo, nothing slower | −25% (93 of 368, on the v1 chain); dict −4%, others ±2% |
