tobogganSe connecter

packages

The Louise packages, each released on its own

Clone avec HTTPS, un jeton personnel en mot de passe, ou avec SSH, une clé ajoutée dans vos paramètres.

239 lignes · 10947 octetsBrut · Historique

fetch

An HTTP/1.1 client in pure Louise for fetching what a URL points to: one GET, over TCP or TLS, guarded against server-side request forgery (SSRF). Built for link previews; depends on packages/url (ADR 0079).

Needs engineering review before production use: it is an SSRF surface. ADR 0079 lists the open questions.

import fetch.{Options, Response}

use config <- result.try (tls.config (tls.defaults ()))   // once, and share it
use page <- result.try (url.parse "https://example.com/post")
let opts = Options(..fetch.options config, max_body: 2097152)
match fetch.get page [("User-Agent", "toki/1")] opts
{ Ok(Response(200, headers, body, truncated)) -> ...
, Ok(Response(status, ..)) -> ...   // 3xx included: redirects are not followed
, Error(e) -> ...
}

API

pub type Policy { PublicOnly | AnyAddress }
pub type Options { Options(tls: Resource<TlsConfig>, policy: Policy,
                           max_body: Int, max_head: Int, deadline: Int) }
pub fn options :: config: Resource<TlsConfig> -> Options / <clock>
pub fn deadline_in :: ms: Int -> Int / <clock>

pub type Response { Response(status: Int, headers: List<(String, String)>,
                             body: Bytes, truncated: Bool) }
pub fn header :: resp: Response name: String -> Option<String>

pub type Error
{ BadUrl(String) | BadHeader(String) | Refused(host: String, ip: String)
| NoAddress(String) | Connect(String) | Tls(TlsError) | Malformed(String) | TimedOut }

pub fn get :: target: Url headers: List<(String, String)> opts: Options
  -> Result<Response, Error> / <proc, clock>

pub fn request :: method: String target: Url headers: List<(String, String)>
  body: Bytes opts: Options -> Result<Response, Error> / <proc, clock>

pub type PendingRequest
pub type RequestOutcome { Completed(Result<Response, Error>) | Cancelled }
pub fn start_request :: method: String target: Url headers: List<(String, String)>
  body: Bytes opts: Options -> Result<PendingRequest, Error> / <proc, clock>
pub fn await_request :: PendingRequest -> RequestOutcome / <proc, clock>
pub fn cancel_request :: PendingRequest -> () / <proc>
  • request is get with any method and a body (ADR 0079, Revision 1): fetch.request "POST" api [("Content-Type", "application/json")] body opts. The body is sent with its Content-Length (a POST or PUT without one says 0); the caller's own Content-Length is dropped with the other framing headers. The method must be a token. get is request "GET" … (bytes.empty ()).

  • options config is PublicOnly, a 1 MiB body, a 64 KiB head and a deadline 10 seconds from now. Change it with a record update.

  • deadline is an instant of time.monotonic, in milliseconds: deadline_in 10000. One deadline can cover a whole redirect chain.

  • header ignores case and gives the first value.

Cancellable collected requests

start_request starts the same checked exchange as request and returns its handle instead of waiting. await_request collects the head and body, returning Completed with the ordinary Result, or Cancelled. request and get keep their blocking behavior.

The starter owns the handle's reply and monitor subjects and must call await_request exactly once, including after cancellation. Any actor holding the handle may call cancel_request: it stops only this exchange, closes its connection as the exchange actor unwinds, and is harmless before, during, or after completion. The starting actor's await_request removes the monitor and discards a reply racing with the kill. A response completed before cancellation can still be Completed; work killed before its reply cannot become a late success.

The deadline covers time between starting and awaiting as well as the wait, and still returns Completed(Error(TimedOut)). As with the deadline of request, a kill that lands inside net.resolve takes effect when the lookup returns; the caller does not wait for it.

For example, a coordinator can share the handle with a stop actor:

use pending <- result.try (fetch.start_request "POST" api headers body opts)
// Send pending to the actor responsible for stop/navigation too.
match fetch.await_request pending
{ Completed(answer) -> use_answer answer
, Cancelled -> reader_left ()
}

The stop actor calls fetch.cancel_request pending. This does not cancel other requests to the same URL or host.

What get does

  1. Checks before any connection. Only http and https. The URL is parsed again from its text, so a hand-built Url gets the same checks as a parsed one. A header name must be a token and a value must not hold a CR, LF or NUL (BadHeader).
  2. Resolves once, checks every address. net.resolve gives the host's addresses. Under PublicOnly, if any of them is not public (fetch/address), the fetch is Refused(host, ip). Then it connects to those same addresses, in order, until one accepts: the name is not resolved again, so a second lookup cannot answer differently (DNS rebinding).
  3. TLS for https, with the caller's configuration. The host name is the server name sent (SNI) and the name the certificate must have.
  4. Sends GET <path?query> HTTP/1.1, Host (with the port when it is not the scheme's), Connection: close, Accept-Encoding: identity (there is no decompression), then the caller's headers. The caller's Host, Connection, Accept-Encoding, Content-Length, Transfer-Encoding, TE and Upgrade are dropped.
  5. Reads the head (at most max_head bytes), skipping 1xx responses, then the body by Content-Length, chunked, or until the server closes. Another Transfer-Encoding, disagreeing Content-Lengths, or a connection that ends before the body does are Malformed. A TLS stream that ends without close_notify ends a body read until close.
  6. Keeps at most max_body bytes and stops reading there, with truncated: True.
  7. Never follows redirects. A 3xx is returned as it is. To follow one, resolve Location with url.resolve and call get again with the same deadline: the new host is checked like the first.
  8. Deadline. The exchange runs in an actor of its own. At the deadline the caller kills it and answers TimedOut, wherever it was waiting. A kill that lands while the actor waits in net.resolve (on the io blocking pool) takes effect when the lookup returns; the caller does not wait for that.

Streamed responses

For server-sent events and other long answers, request_streaming returns once the response head has arrived. You then read the body as it comes (ADR 0079, Revision 2):

pub type Streamed { Streamed(status: Int, headers: List<(String, String)>, body: BodyStream) }
pub type Piece { Data(Bytes) | End(truncated: Bool) }

pub fn request_streaming :: method: String target: Url headers: List<(String, String)>
  body: Bytes opts: Options -> Result<Streamed, Error> / <proc, clock>
pub fn next :: stream: BodyStream -> Result<Piece, Error> / <proc, clock>
pub fn cancel :: stream: BodyStream -> () / <proc>
  • Same checks as request: the URL, the headers, the address policy and the limits. A failure before the head is the Error of request_streaming.
  • Reading: next gives the body's bytes as they arrive (a chunked body already decoded), then End. After End or an Error the connection is closed.
  • Deadline: opts.deadline covers the whole exchange, body included, so give a long answer a long deadline. At the deadline next answers TimedOut and stops the exchange.
  • Cancelling: cancel closes the connection, so the server sees the client leave, and drops what arrived unread. Call it when you stop reading early. It is harmless after the end.
  • Memory: at most max_body bytes are delivered, so a reader that falls behind holds at most that much in its mailbox.
  • Ownership: only the actor that made the request can read the stream.

fetch/sse parses server-sent events (text/event-stream) from those pieces:

import sse
let (parser, events) = sse.feed parser piece   // [Event(name: "delta", data: "Bon", id: "")]

It follows the HTML standard's rules:

  • lines end with CRLF, LF or CR, even when one is split across pieces;
  • a blank line dispatches an event that has data;
  • event, data and id are read, and the id persists from one event to the next;
  • retry is readable with sse.retry;
  • comments and unknown fields are ignored;
  • a leading byte order mark is dropped.

Reconnecting is the caller's choice.

fetch/address

classify ip -> Result<Reach, String> with Reach { Public | Private(reason) }, and is_public ip. Refused ranges:

IPv4IPv6
0.0.0.0/8 this network:: unspecified, ::1 loopback
10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 privatefc00::/7 unique local
100.64.0.0/10 carrier-grade NATfe80::/10 link-local, fec0::/10 site-local
127.0.0.0/8 loopbackff00::/8 multicast
169.254.0.0/16 link-local (cloud metadata)::/96 IPv4-compatible
224.0.0.0 and up: multicast, reserved, broadcast

::ffff:a.b.c.d (IPv4-mapped) and 64:ff9b::/96 (NAT64) are judged by the IPv4 address they hold. Only the forms net.resolve gives are read: anything else (leading zeros, a zone, a host name) is an Error, and PublicOnly refuses it.

fetch/response

The pure parsers get uses, tested on their own: head data max_head (Ok(None) while incomplete), framing head, and a chunked decoder fed as bytes arrive (dechunker max, feed, is_done, body, and take, which hands out what was decoded so far for a streamed body).

Tests

The parsers have property tests in both back ends. The loopback tests (std/http, a raw TCP server, a TLS server with the fixtures of test/fixtures/tls) need sockets, which only native code has, so they run when LOUISE_NET_TESTS is set:

LOUISE_NET_TESTS=1 louise test --native

The streamed responses' tests (test/stream_test.lse) use a raw server that writes a response in steps the test controls. They check that:

  • a piece arrives before the rest of the body is sent;
  • the cap applies to a body by length and to one read until close;
  • the deadline stops a stalled body;
  • cancel closes the connection, as the server sees it;
  • failures before the head are errors.

The pending-request tests use similarly controlled loopback servers to check cancellation before the response, during a partial body and during a TLS handshake; observed connection closes; repeated cancellation and late answers; successful completion and deadlines. Repeated exchanges exercise cleanup across fresh handles and PIDs.

The SSE parser's tests, including a property that any split of a stream parses the same, run in both back ends.