# fetch An HTTP/1.1 client in pure Louise for fetching what a URL points to: one GET, over TCP or TLS, guarded against server-side request forgery (SSRF). Built for link previews; depends on `packages/url` (ADR 0079). **Needs engineering review before production use**: it is an SSRF surface. ADR 0079 lists the open questions. ```louise import fetch.{Options, Response} use config <- result.try (tls.config (tls.defaults ())) // once, and share it use page <- result.try (url.parse "https://example.com/post") let opts = Options(..fetch.options config, max_body: 2097152) match fetch.get page [("User-Agent", "toki/1")] opts { Ok(Response(200, headers, body, truncated)) -> ... , Ok(Response(status, ..)) -> ... // 3xx included: redirects are not followed , Error(e) -> ... } ``` ## API ```louise pub type Policy { PublicOnly | AnyAddress } pub type Options { Options(tls: Resource, policy: Policy, max_body: Int, max_head: Int, deadline: Int) } pub fn options :: config: Resource -> Options / pub fn deadline_in :: ms: Int -> Int / pub type Response { Response(status: Int, headers: List<(String, String)>, body: Bytes, truncated: Bool) } pub fn header :: resp: Response name: String -> Option pub type Error { BadUrl(String) | BadHeader(String) | Refused(host: String, ip: String) | NoAddress(String) | Connect(String) | Tls(TlsError) | Malformed(String) | TimedOut } pub fn get :: target: Url headers: List<(String, String)> opts: Options -> Result / pub fn request :: method: String target: Url headers: List<(String, String)> body: Bytes opts: Options -> Result / pub type PendingRequest pub type RequestOutcome { Completed(Result) | Cancelled } pub fn start_request :: method: String target: Url headers: List<(String, String)> body: Bytes opts: Options -> Result / pub fn await_request :: PendingRequest -> RequestOutcome / pub fn cancel_request :: PendingRequest -> () / ``` - `request` is `get` with any method and a body (ADR 0079, Revision 1): `fetch.request "POST" api [("Content-Type", "application/json")] body opts`. The body is sent with its `Content-Length` (a POST or PUT without one says `0`); the caller's own `Content-Length` is dropped with the other framing headers. The method must be a token. `get` is `request "GET" … (bytes.empty ())`. - `options config` is `PublicOnly`, a 1 MiB body, a 64 KiB head and a deadline 10 seconds from now. Change it with a record update. - `deadline` is an instant of `time.monotonic`, in milliseconds: `deadline_in 10000`. One deadline can cover a whole redirect chain. - `header` ignores case and gives the first value. ## Cancellable collected requests `start_request` starts the same checked exchange as `request` and returns its handle instead of waiting. `await_request` collects the head and body, returning `Completed` with the ordinary `Result`, or `Cancelled`. `request` and `get` keep their blocking behavior. The starter owns the handle's reply and monitor subjects and must call `await_request` exactly once, including after cancellation. Any actor holding the handle may call `cancel_request`: it stops only this exchange, closes its connection as the exchange actor unwinds, and is harmless before, during, or after completion. The starting actor's `await_request` removes the monitor and discards a reply racing with the kill. A response completed before cancellation can still be `Completed`; work killed before its reply cannot become a late success. The deadline covers time between starting and awaiting as well as the wait, and still returns `Completed(Error(TimedOut))`. As with the deadline of `request`, a kill that lands inside `net.resolve` takes effect when the lookup returns; the caller does not wait for it. For example, a coordinator can share the handle with a stop actor: ```louise use pending <- result.try (fetch.start_request "POST" api headers body opts) // Send pending to the actor responsible for stop/navigation too. match fetch.await_request pending { Completed(answer) -> use_answer answer , Cancelled -> reader_left () } ``` The stop actor calls `fetch.cancel_request pending`. This does not cancel other requests to the same URL or host. ## What `get` does 1. **Checks before any connection.** Only `http` and `https`. The URL is parsed again from its text, so a hand-built `Url` gets the same checks as a parsed one. A header name must be a token and a value must not hold a CR, LF or NUL (`BadHeader`). 2. **Resolves once, checks every address.** `net.resolve` gives the host's addresses. Under `PublicOnly`, if **any** of them is not public (`fetch/address`), the fetch is `Refused(host, ip)`. Then it connects to those same addresses, in order, until one accepts: the name is not resolved again, so a second lookup cannot answer differently (DNS rebinding). 3. **TLS** for `https`, with the caller's configuration. The host name is the server name sent (SNI) and the name the certificate must have. 4. **Sends** `GET HTTP/1.1`, `Host` (with the port when it is not the scheme's), `Connection: close`, `Accept-Encoding: identity` (there is no decompression), then the caller's headers. The caller's `Host`, `Connection`, `Accept-Encoding`, `Content-Length`, `Transfer-Encoding`, `TE` and `Upgrade` are dropped. 5. **Reads** the head (at most `max_head` bytes), skipping `1xx` responses, then the body by `Content-Length`, `chunked`, or until the server closes. Another `Transfer-Encoding`, disagreeing `Content-Length`s, or a connection that ends before the body does are `Malformed`. A TLS stream that ends without `close_notify` ends a body read until close. 6. **Keeps at most `max_body` bytes** and stops reading there, with `truncated: True`. 7. **Never follows redirects.** A 3xx is returned as it is. To follow one, resolve `Location` with `url.resolve` and call `get` again with the same deadline: the new host is checked like the first. 8. **Deadline.** The exchange runs in an actor of its own. At the deadline the caller kills it and answers `TimedOut`, wherever it was waiting. A kill that lands while the actor waits in `net.resolve` (on the `io` blocking pool) takes effect when the lookup returns; the caller does not wait for that. ## Streamed responses For server-sent events and other long answers, `request_streaming` returns once the response head has arrived. You then read the body as it comes (ADR 0079, Revision 2): ```louise pub type Streamed { Streamed(status: Int, headers: List<(String, String)>, body: BodyStream) } pub type Piece { Data(Bytes) | End(truncated: Bool) } pub fn request_streaming :: method: String target: Url headers: List<(String, String)> body: Bytes opts: Options -> Result / pub fn next :: stream: BodyStream -> Result / pub fn cancel :: stream: BodyStream -> () / ``` - **Same checks as `request`:** the URL, the headers, the address policy and the limits. A failure before the head is the `Error` of `request_streaming`. - **Reading:** `next` gives the body's bytes as they arrive (a chunked body already decoded), then `End`. After `End` or an `Error` the connection is closed. - **Deadline:** `opts.deadline` covers the whole exchange, body included, so give a long answer a long deadline. At the deadline `next` answers `TimedOut` and stops the exchange. - **Cancelling:** `cancel` closes the connection, so the server sees the client leave, and drops what arrived unread. Call it when you stop reading early. It is harmless after the end. - **Memory:** at most `max_body` bytes are delivered, so a reader that falls behind holds at most that much in its mailbox. - **Ownership:** only the actor that made the request can read the stream. `fetch/sse` parses server-sent events (`text/event-stream`) from those pieces: ```louise import sse let (parser, events) = sse.feed parser piece // [Event(name: "delta", data: "Bon", id: "")] ``` It follows the HTML standard's rules: - lines end with CRLF, LF or CR, even when one is split across pieces; - a blank line dispatches an event that has data; - `event`, `data` and `id` are read, and the `id` persists from one event to the next; - `retry` is readable with `sse.retry`; - comments and unknown fields are ignored; - a leading byte order mark is dropped. Reconnecting is the caller's choice. ## `fetch/address` `classify ip -> Result` with `Reach { Public | Private(reason) }`, and `is_public ip`. Refused ranges: | IPv4 | IPv6 | |---|---| | `0.0.0.0/8` this network | `::` unspecified, `::1` loopback | | `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` private | `fc00::/7` unique local | | `100.64.0.0/10` carrier-grade NAT | `fe80::/10` link-local, `fec0::/10` site-local | | `127.0.0.0/8` loopback | `ff00::/8` multicast | | `169.254.0.0/16` link-local (cloud metadata) | `::/96` IPv4-compatible | | `224.0.0.0` and up: multicast, reserved, broadcast | | `::ffff:a.b.c.d` (IPv4-mapped) and `64:ff9b::/96` (NAT64) are judged by the IPv4 address they hold. Only the forms `net.resolve` gives are read: anything else (leading zeros, a zone, a host name) is an `Error`, and `PublicOnly` refuses it. ## `fetch/response` The pure parsers `get` uses, tested on their own: `head data max_head` (`Ok(None)` while incomplete), `framing head`, and a chunked decoder fed as bytes arrive (`dechunker max`, `feed`, `is_done`, `body`, and `take`, which hands out what was decoded so far for a streamed body). ## Tests The parsers have property tests in both back ends. The loopback tests (std/http, a raw TCP server, a TLS server with the fixtures of `test/fixtures/tls`) need sockets, which only native code has, so they run when `LOUISE_NET_TESTS` is set: ```sh LOUISE_NET_TESTS=1 louise test --native ``` The streamed responses' tests (`test/stream_test.lse`) use a raw server that writes a response in steps the test controls. They check that: - a piece arrives before the rest of the body is sent; - the cap applies to a body by length and to one read until close; - the deadline stops a stalled body; - `cancel` closes the connection, as the server sees it; - failures before the head are errors. The pending-request tests use similarly controlled loopback servers to check cancellation before the response, during a partial body and during a TLS handshake; observed connection closes; repeated cancellation and late answers; successful completion and deadlines. Repeated exchanges exercise cleanup across fresh handles and PIDs. The SSE parser's tests, including a property that any split of a stream parses the same, run in both back ends.