| @@ -0,0 +1,76 @@ |
| 1 | +# ADR 0061: A fast test loop |
| 2 | + |
| 3 | +**Status:** Proposed (2026-10-24, issue #316); needs engineering review (#44), in particular the result cache's key and what it does not track ([Kept results](#kept-results)), and interpreted tests skipping the optimiser (ADR 0043's note of the same date). |
| 4 | + |
| 5 | +## Context |
| 6 | + |
| 7 | +On toki2 (6.2k lines, 372 tests, 22 cores), `louise test` took about 31 s on every run, on one core, whether anything had changed or not. Checking the same package took 0.26 s. The time went to: |
| 8 | + |
| 9 | +- **5.7 s before the first test:** lowering, then the optimiser (about 4.5 s), for every test's entry at once; |
| 10 | +- **23 s of tests, one after another:** a median of 57 ms each, even for a test comparing two strings. Each test deep-copied the whole optimised program (4,292 functions) before it ran. |
| 11 | + |
| 12 | +`louise test --native` took 11 s every run: tests ran one process at a time, and the test executable was generated, compiled and linked again each time. A dev build after a one-line change took 5.2 s, the C units compiled in batches that waited for their slowest. `build/` kept the units of deleted modules for good. For test-driven work this is the loop that matters. `go test` and `cargo test` answer a package this size in a second or two, and repeat nothing that has not changed. |
| 13 | + |
| 14 | +## Decision |
| 15 | + |
| 16 | +**Tests share one program.** `louise_interp::run` takes anything that is an `Arc<Program>`; the interpreter never changes a program, so tests running at once share it. This alone takes toki2's tests from 23 s to 2.4 s. |
| 17 | + |
| 18 | +**Tests run in parallel and report in order.** `louise test` runs tests on a work queue, `--jobs N` at once (default: one per core), interpreted and native alike. Each test's stdout and stderr are captured and printed with its result line in the tests' order, so a parallel run prints what a serial one does (`--jobs 1`). The interpreter takes `RunOptions`: |
| 19 | + |
| 20 | +- an error writer for `io.eprintln` and a crashed actor's report; |
| 21 | +- `exit_ends_run`, with which `os.exit` ends that run (`RtError::Exit`) instead of the runner: a test calling it fails, and the others still run; |
| 22 | +- `inputs`, where a run notes what it reads from outside the program (below). |
| 23 | + |
| 24 | +Native test processes share the cores: `LOUISE_THREADS` defaults to cores ÷ jobs when unset. |
| 25 | + |
| 26 | +**`--filter <text>`** runs the tests whose `module.name` contains the text, before anything is optimised or built. |
| 27 | + |
| 28 | +**Interpreted tests run unoptimised Core** (TRMC still runs); `--opt` runs the optimised one. The optimiser is checked where it matters: every `tests/run` program runs both ways, native tests run optimised Core, and `--opt` checks a package's own tests (ADR 0043, note of 2026-10-24). |
| 29 | + |
| 30 | +**The optimiser is faster for everyone,** with byte-identical output (the md5 of `louise core` over every `tests/run` program, example and toki2): no copies to inspect a block or to try case-of-case before its cheap conditions hold, and ids hashed with one multiply instead of SipHash (`louise_core::idhash`). On toki2: 23.0 G instructions to 13.9 G. |
| 31 | + |
| 32 | +**`--timings`** reports a test run's phases, `optimise` and `run` included. |
| 33 | + |
| 34 | +### Kept results |
| 35 | + |
| 36 | +A test that passed is not run again while nothing it depends on changed, as `go test` does; it reports `ok (cached)`. Interpreted runs only: what a native test process reads is not recorded. |
| 37 | + |
| 38 | +- **The key** covers the compiler (its executable's size and modification time, as the build fingerprint does), the mode (`--opt` or not), the `LOUISE_*` environment, the working directory, the manifest, the test's name, and the text of every module the test's module imports, transitively: the prelude, std's and dependencies' modules included. |
| 39 | +- **What the run read** is kept with the pass and checked again: each environment variable and its value, and a fingerprint of each file or directory read, listed or looked for (its contents, its entries, or that it is absent). |
| 40 | +- **Not kept:** a failure; a run that wrote a file, read its arguments or the runtime's pool sizes, which no later run can check. |
| 41 | +- **Not tracked:** the clock and randomness. A test whose verdict depends on them is flaky, and rerunning it is not what makes it pass. `go test` makes the same choice. |
| 42 | +- **Where:** `build/cache/tests.<mode>`, read and written under the build directory's lock and merged, so concurrent runs keep each other's results. `--no-cache` or `LOUISE_TEST_CACHE=off` runs everything. |
| 43 | + |
| 44 | +When every selected test is kept, nothing is optimised or run. After an edit, only the tests that import the edited module (or read an edited file) run: the affected tests, with no separate dependency analysis. |
| 45 | + |
| 46 | +**`--watch`** runs the tests, then again after each change to the package's `.lse` files (new ones included), `louise.toml`, `louise.lock`, or a source the last run read (path dependencies). It polls sizes and modification times every 300 ms: no new dependency. |
| 47 | + |
| 48 | +### Builds |
| 49 | + |
| 50 | +**One work queue** replaces the batches of `louise_build::in_parallel`: a worker takes the next job as soon as it is done with one (`each_in_parallel` streams results to the caller, for the test runner). **The native test executable is fingerprinted** like `louise build`'s, with the tests it runs in the key: an unchanged `test --native` generates and links nothing. **A build removes the units it no longer writes:** `cache/<name>.<profile>.units` records them, and the next build removes their `.c`, `.o`, `.d` and `.key` (only the record decides; a name prefix cannot tell `foo` + `bin/x` from `foo.bin` + `x`). Leftovers from before the record are not touched. |
| 51 | + |
| 52 | +## Consequences |
| 53 | + |
| 54 | +toki2, before and after (22 cores, the machine otherwise busy): |
| 55 | + |
| 56 | +| | Before | After | |
| 57 | +|---|---:|---:| |
| 58 | +| `louise test`, first run | 30.7 s | 1.3 s | |
| 59 | +| `louise test`, nothing changed | 30.9 s | 0.4 s (370 kept, 2 read the runtime's pools) | |
| 60 | +| `louise test` after editing `notes/web/paths` | 30.6 s | 0.7 s (81 run, 291 kept) | |
| 61 | +| `louise test --opt` | — | 6.4 s | |
| 62 | +| `louise test --native`, nothing changed | 11.1 s | 0.9 s | |
| 63 | +| `louise core` (the optimiser) | 23.0 G instr. | 13.9 G instr. | |
| 64 | +| dev build, cold | 11.4 s | 9.2 s | |
| 65 | + |
| 66 | +- Output is unchanged for a serial run; a parallel one prints the same. |
| 67 | +- A test sharing state outside the program with another (a file both write, a database) may now race: `--jobs 1` runs them one at a time. |
| 68 | +- A rerun reports kept passes as `ok (cached)`; tools that compare outputs between runs turn the cache off (`crates/louise/tests/cli.rs` does). |
| 69 | +- Not done: caching the optimised Core between runs. Lowering is 0.1–0.5 s once interpreted tests skip the optimiser, and kept results cover the unchanged case; serialising Core (no serde today, `FnId`s shift with every edit) would cost a format to maintain for at most that. |
| 70 | + |
| 71 | +## Alternatives considered |
| 72 | + |
| 73 | +- **Parallel tests on one interpreter thread pool, sharing values:** the interpreter's values are `Rc`; per-run threads keep them apart for free. |
| 74 | +- **Watching with `notify`/inotify:** a dependency and per-platform behaviour for a loop that polls a few hundred files cheaply. |
| 75 | +- **Affected tests from a separate dependency analysis:** the result key already holds each test's import closure; a kept result *is* "not affected". |
| 76 | +- **Recording the clock and randomness as inputs:** every test that touches time would rerun every time, which is most of a web application's tests. |