| @@ -0,0 +1,143 @@ |
| 1 | +# ADR 0079: URLs and an HTTP client are packages, `packages/url` and `packages/fetch`, and a fetch refuses private addresses |
| 2 | + |
| 3 | +**Status:** Proposed (2026-11-07); needs engineering review (#44): the |
| 4 | +client is a server-side request forgery (SSRF) surface. |
| 5 | + |
| 6 | +## Context |
| 7 | + |
| 8 | +- **toki2 wants link previews.** A note that holds a link shows a card |
| 9 | + with the page's title, description and image. The server has to fetch |
| 10 | + a URL a user typed, then maybe the image that page names. |
| 11 | +- **Louise has no HTTP client.** It has TCP (`std/net`), TLS |
| 12 | + (`std/tls`, ADR 0038), `net.resolve` and an HTTP *server*. There is no |
| 13 | + URL parser either: toki2 would have to split URLs by hand to resolve a |
| 14 | + page's relative image path. |
| 15 | +- **A server that fetches what users name is an SSRF surface.** A URL can |
| 16 | + point at `127.0.0.1`, the private network, or the cloud's metadata |
| 17 | + service (`169.254.169.254`). A host can resolve to a public address |
| 18 | + for the check and a private one for the connection (DNS rebinding). A |
| 19 | + redirect can lead from a public host to a private one. A slow or |
| 20 | + endless server can hold a worker forever. |
| 21 | +- toki (the earlier app) blocked these ranges in Node; toki2 needs the |
| 22 | + same guard, and it belongs next to the client, not in each app. |
| 23 | + |
| 24 | +## Decision |
| 25 | + |
| 26 | +- **Two packages, not std**, as for `json` and `log` (ADR 0075, 0077): |
| 27 | + pure Louise, depended on by a git tag. Their APIs are new and will |
| 28 | + change; std is versioned with the compiler. |
| 29 | +- **`packages/url`** (RFC 3986), for URLs with a host: |
| 30 | + - `Url(scheme, host, port: Option<Int>, path, query: Option<String>, |
| 31 | + fragment: Option<String>)`, `parse`, `resolve base reference` |
| 32 | + (§5.2, with `remove_dot_segments`), `to_string`, `port_or_default` |
| 33 | + and `request_target`. |
| 34 | + - The scheme and host are lowercased, an IPv6 host keeps its brackets, |
| 35 | + the path loses its dot segments. `parse (to_string u) == Ok(u)`. |
| 36 | + - **Refused:** userinfo (`user:pass@host`, which hides the real host |
| 37 | + from a reader), a host that is not ASCII (**no IDNA**: callers send |
| 38 | + the `xn--` form), a host character outside `A-Z a-z 0-9 - . _ ~`, a |
| 39 | + port above 65535, and any control character. |
| 40 | + - `request_target` percent-encodes every byte that may not appear in |
| 41 | + a request line, so even a hand-built `Url` cannot inject a header. |
| 42 | + - A URL without a host (`mailto:`, the RFC's `g:h`) is not |
| 43 | + represented: `resolve` refuses it. |
| 44 | +- **`packages/fetch`**, depending on `url`: |
| 45 | + - `fetch/address`: `classify ip -> Result<Reach, String>`, |
| 46 | + `Reach { Public | Private(reason) }`, `is_public`. Refused: IPv4 |
| 47 | + `0/8`, `10/8`, `100.64/10`, `127/8`, `169.254/16`, `172.16/12`, |
| 48 | + `192.168/16`, `224/4` and `240/4` (with the broadcast address); IPv6 |
| 49 | + `::`, `::1`, `fe80::/10`, `fec0::/10`, `fc00::/7`, `ff00::/8` and |
| 50 | + the rest of `::/96`. `::ffff:a.b.c.d` and `64:ff9b::/96` (NAT64) are |
| 51 | + judged by the IPv4 address they hold. Text that is not an address |
| 52 | + as `net.resolve` writes it is an `Error`, and refused. |
| 53 | + - `fetch/response`: the pure parsers (head, framing, a chunked decoder |
| 54 | + fed as bytes arrive), property-tested on their own. |
| 55 | + - `fetch.get target headers Options(tls, policy, max_body, max_head, |
| 56 | + deadline) -> Result<Response(status, headers, body, truncated), |
| 57 | + Error>`, where `Error` is `BadUrl | BadHeader | Refused(host, ip) | |
| 58 | + NoAddress | Connect | Tls(TlsError) | Malformed | TimedOut`. |
| 59 | +- **How `get` guards the fetch:** |
| 60 | + - **Every address is checked.** The host is resolved once. Under |
| 61 | + `PublicOnly` the fetch is refused if *any* address is private, not |
| 62 | + only the one connected to. Then the client connects to those checked |
| 63 | + addresses, in order, and never resolves the name again, so a second |
| 64 | + answer cannot differ from the checked one (rebinding). |
| 65 | + - **TLS** uses the host name for SNI and certificate verification |
| 66 | + while connecting to the checked IP. |
| 67 | + - **No redirects.** A 3xx is returned as it is. The caller resolves |
| 68 | + `Location` with `url.resolve` and calls `get` again, which checks the |
| 69 | + new host from the start; the caller also bounds the chain. |
| 70 | + - **Headers cannot be injected.** A name must be a token and a value |
| 71 | + must not hold CR, LF or NUL. The client owns the framing headers: the |
| 72 | + caller's `Host`, `Connection`, `Accept-Encoding`, `Content-Length`, |
| 73 | + `Transfer-Encoding`, `TE` and `Upgrade` are dropped. |
| 74 | + - **Bounded in size.** The head is capped (`max_head`, 64 KiB by |
| 75 | + default) and the body is kept to `max_body` bytes (1 MiB), with |
| 76 | + `truncated` saying so; reading stops there. |
| 77 | + - **Bounded in time.** The whole exchange (lookup, connect, handshake, |
| 78 | + reads) runs in an actor started with `spawn_monitor`; the caller |
| 79 | + waits with `select` until `deadline` (a `time.monotonic` instant, so |
| 80 | + one deadline covers a redirect chain), then kills it and answers |
| 81 | + `TimedOut`. |
| 82 | + - `Accept-Encoding: identity`: Louise has no inflate, so nothing |
| 83 | + compressed is asked for, and a response with a `Transfer-Encoding` |
| 84 | + other than `chunked` is `Malformed`. |
| 85 | +- **Tests:** property tests for the parsers (URL round trips, the |
| 86 | + RFC 3986 §5.4 tables, address ranges against a reference written with |
| 87 | + 32-bit numbers, chunked encoding split at random points and cut at |
| 88 | + random limits), in both back ends. Loopback tests (std/http, a raw TCP |
| 89 | + server, a TLS server with the fixtures' private CA) run natively when |
| 90 | + `LOUISE_NET_TESTS` is set, as `LOUISE_PG_URL` gates the PostgreSQL |
| 91 | + tests. |
| 92 | + |
| 93 | +## Consequences |
| 94 | + |
| 95 | +- toki2 can fetch previews with `PublicOnly`, a few redirects re-checked |
| 96 | + by itself, and its own caps. Tests use `AnyAddress` against |
| 97 | + `127.0.0.1`. |
| 98 | +- **A kill while the worker waits in `net.resolve`** lands when the |
| 99 | + lookup returns: a blocking call keeps its actor alive until it ends |
| 100 | + (`runtime/louise_rt/src/blocking.rs`). `get` does not wait for that; it |
| 101 | + answers `TimedOut` at the deadline. But each timed-out lookup holds an |
| 102 | + `io` pool thread until the resolver gives up, so many hung lookups can |
| 103 | + queue the next ones (`runtime.blocking_stats` shows it). The loopback |
| 104 | + test checks that a 1 ms deadline against a name that does not resolve |
| 105 | + answers on time; it cannot force the kill to land inside the lookup. |
| 106 | +- **A body read until close** cannot tell a complete body from one cut by |
| 107 | + the network (over TLS, a missing `close_notify` is taken as the end). |
| 108 | + `Content-Length` and chunked bodies that end early are `Malformed`. |
| 109 | +- **One connection per fetch** (`Connection: close`): no keep-alive, no |
| 110 | + pooling, no HTTP/2. Enough for previews. |
| 111 | +- CI checks and tests both packages with the others (`scripts/ci.sh`, |
| 112 | + packages part); the loopback tests run where `LOUISE_NET_TESTS` is set. |
| 113 | + |
| 114 | +## Open questions for the engineering review |
| 115 | + |
| 116 | +- **Ports.** Any port is allowed. Should `PublicOnly` also allow only 80 |
| 117 | + and 443, so a public host cannot be used to probe other services? |
| 118 | +- **More ranges.** Not refused today: `192.0.0.0/24`, the documentation |
| 119 | + nets (`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`), |
| 120 | + `198.18.0.0/15` (benchmarking), 6to4 (`2002::/16`, which embeds an IPv4 |
| 121 | + address), Teredo (`2001::/32`) and the local-use NAT64 prefix |
| 122 | + `64:ff9b:1::/48`. |
| 123 | +- **DNS pinning.** The checked addresses are the ones connected to, |
| 124 | + within one fetch. Whether that is enough against rebinding across a |
| 125 | + redirect chain (each `get` resolves again) is for the review. |
| 126 | +- **The resolver.** A hung lookup holds an `io` thread past the |
| 127 | + deadline (above). A lookup with its own timeout would need the runtime. |
| 128 | +- **IDNA.** Non-ASCII hosts are refused rather than converted. |
| 129 | + |
| 130 | +## Alternatives considered |
| 131 | + |
| 132 | +- **Follow redirects inside `get`.** Simpler for callers, but the policy |
| 133 | + for how many, and which schemes, belongs to them, and a redirect must |
| 134 | + be checked like the first host anyway. Returning the 3xx keeps `get` |
| 135 | + one request. |
| 136 | +- **Check only the address connected to.** A host with one public and one |
| 137 | + private address would pass on a lucky order. Refusing the host when any |
| 138 | + address is private is stricter and simpler to reason about. |
| 139 | +- **Resolve, check, then `net.connect host`.** `net.connect` resolves |
| 140 | + again, which reopens the rebinding window. Connecting to the checked IPs |
| 141 | + closes it. |
| 142 | +- **Timeouts on each socket operation.** std has none (ADR 0038 leaves |
| 143 | + them to the caller); a killed worker covers every wait at once. |