tobogganSe connecter

packages

The Louise packages, each released on its own

Clone avec HTTPS, un jeton personnel en mot de passe, ou avec SSH, une clé ajoutée dans vos paramètres.

NomDernier commitDate
src/Fetch: streamed responses and a server-sent events parseril y a 6 j
test/A repository of its own: README, CI, release checks, fixturesil y a 4 j
CHANGELOG.mdA repository of its own: README, CI, release checks, fixturesil y a 4 j
louise.tomlVersion every package 0.21.0, on louise 0.22.0il y a 4 j
README.mdA repository of its own: README, CI, release checks, fixturesil y a 4 j
README.MD

fetch

An HTTP/1.1 client in pure Louise for fetching what a URL points to: one GET, over TCP or TLS, guarded against server-side request forgery (SSRF). Built for link previews; depends on packages/url (ADR 0079).

Needs engineering review before production use: it is an SSRF surface. ADR 0079 lists the open questions.

import fetch.{Options, Response}

use config <- result.try (tls.config (tls.defaults ()))   // once, and share it
use page <- result.try (url.parse "https://example.com/post")
let opts = Options(..fetch.options config, max_body: 2097152)
match fetch.get page [("User-Agent", "toki/1")] opts
{ Ok(Response(200, headers, body, truncated)) -> ...
, Ok(Response(status, ..)) -> ...   // 3xx included: redirects are not followed
, Error(e) -> ...
}

API

pub type Policy { PublicOnly | AnyAddress }
pub type Options { Options(tls: Resource<TlsConfig>, policy: Policy,
                           max_body: Int, max_head: Int, deadline: Int) }
pub fn options :: config: Resource<TlsConfig> -> Options / <clock>
pub fn deadline_in :: ms: Int -> Int / <clock>

pub type Response { Response(status: Int, headers: List<(String, String)>,
                             body: Bytes, truncated: Bool) }
pub fn header :: resp: Response name: String -> Option<String>

pub type Error
{ BadUrl(String) | BadHeader(String) | Refused(host: String, ip: String)
| NoAddress(String) | Connect(String) | Tls(TlsError) | Malformed(String) | TimedOut }

pub fn get :: target: Url headers: List<(String, String)> opts: Options
  -> Result<Response, Error> / <proc, clock>

pub fn request :: method: String target: Url headers: List<(String, String)>
  body: Bytes opts: Options -> Result<Response, Error> / <proc, clock>
  • request is get with any method and a body (ADR 0079, Revision 1): fetch.request "POST" api [("Content-Type", "application/json")] body opts. The body is sent with its Content-Length (a POST or PUT without one says 0); the caller's own Content-Length is dropped with the other framing headers. The method must be a token. get is request "GET" … (bytes.empty ()).

  • options config is PublicOnly, a 1 MiB body, a 64 KiB head and a deadline 10 seconds from now. Change it with a record update.

  • deadline is an instant of time.monotonic, in milliseconds: deadline_in 10000. One deadline can cover a whole redirect chain.

  • header ignores case and gives the first value.

What get does

  1. Checks before any connection. Only http and https. The URL is parsed again from its text, so a hand-built Url gets the same checks as a parsed one. A header name must be a token and a value must not hold a CR, LF or NUL (BadHeader).
  2. Resolves once, checks every address. net.resolve gives the host's addresses. Under PublicOnly, if any of them is not public (fetch/address), the fetch is Refused(host, ip). Then it connects to those same addresses, in order, until one accepts: the name is not resolved again, so a second lookup cannot answer differently (DNS rebinding).
  3. TLS for https, with the caller's configuration. The host name is the server name sent (SNI) and the name the certificate must have.
  4. Sends GET <path?query> HTTP/1.1, Host (with the port when it is not the scheme's), Connection: close, Accept-Encoding: identity (there is no decompression), then the caller's headers. The caller's Host, Connection, Accept-Encoding, Content-Length, Transfer-Encoding, TE and Upgrade are dropped.
  5. Reads the head (at most max_head bytes), skipping 1xx responses, then the body by Content-Length, chunked, or until the server closes. Another Transfer-Encoding, disagreeing Content-Lengths, or a connection that ends before the body does are Malformed. A TLS stream that ends without close_notify ends a body read until close.
  6. Keeps at most max_body bytes and stops reading there, with truncated: True.
  7. Never follows redirects. A 3xx is returned as it is. To follow one, resolve Location with url.resolve and call get again with the same deadline: the new host is checked like the first.
  8. Deadline. The exchange runs in an actor of its own. At the deadline the caller kills it and answers TimedOut, wherever it was waiting. A kill that lands while the actor waits in net.resolve (on the io blocking pool) takes effect when the lookup returns; the caller does not wait for that.

Streamed responses

For server-sent events and other long answers, request_streaming returns once the response head has arrived. You then read the body as it comes (ADR 0079, Revision 2):

pub type Streamed { Streamed(status: Int, headers: List<(String, String)>, body: BodyStream) }
pub type Piece { Data(Bytes) | End(truncated: Bool) }

pub fn request_streaming :: method: String target: Url headers: List<(String, String)>
  body: Bytes opts: Options -> Result<Streamed, Error> / <proc, clock>
pub fn next :: stream: BodyStream -> Result<Piece, Error> / <proc, clock>
pub fn cancel :: stream: BodyStream -> () / <proc>
  • Same checks as request: the URL, the headers, the address policy and the limits. A failure before the head is the Error of request_streaming.
  • Reading: next gives the body's bytes as they arrive (a chunked body already decoded), then End. After End or an Error the connection is closed.
  • Deadline: opts.deadline covers the whole exchange, body included, so give a long answer a long deadline. At the deadline next answers TimedOut and stops the exchange.
  • Cancelling: cancel closes the connection, so the server sees the client leave, and drops what arrived unread. Call it when you stop reading early. It is harmless after the end.
  • Memory: at most max_body bytes are delivered, so a reader that falls behind holds at most that much in its mailbox.
  • Ownership: only the actor that made the request can read the stream.

fetch/sse parses server-sent events (text/event-stream) from those pieces:

import sse
let (parser, events) = sse.feed parser piece   // [Event(name: "delta", data: "Bon", id: "")]

It follows the HTML standard's rules:

  • lines end with CRLF, LF or CR, even when one is split across pieces;
  • a blank line dispatches an event that has data;
  • event, data and id are read, and the id persists from one event to the next;
  • retry is readable with sse.retry;
  • comments and unknown fields are ignored;
  • a leading byte order mark is dropped.

Reconnecting is the caller's choice.

fetch/address

classify ip -> Result<Reach, String> with Reach { Public | Private(reason) }, and is_public ip. Refused ranges:

IPv4IPv6
0.0.0.0/8 this network:: unspecified, ::1 loopback
10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 privatefc00::/7 unique local
100.64.0.0/10 carrier-grade NATfe80::/10 link-local, fec0::/10 site-local
127.0.0.0/8 loopbackff00::/8 multicast
169.254.0.0/16 link-local (cloud metadata)::/96 IPv4-compatible
224.0.0.0 and up: multicast, reserved, broadcast

::ffff:a.b.c.d (IPv4-mapped) and 64:ff9b::/96 (NAT64) are judged by the IPv4 address they hold. Only the forms net.resolve gives are read: anything else (leading zeros, a zone, a host name) is an Error, and PublicOnly refuses it.

fetch/response

The pure parsers get uses, tested on their own: head data max_head (Ok(None) while incomplete), framing head, and a chunked decoder fed as bytes arrive (dechunker max, feed, is_done, body, and take, which hands out what was decoded so far for a streamed body).

Tests

The parsers have property tests in both back ends. The loopback tests (std/http, a raw TCP server, a TLS server with the fixtures of test/fixtures/tls) need sockets, which only native code has, so they run when LOUISE_NET_TESTS is set:

LOUISE_NET_TESTS=1 louise test --native

The streamed responses' tests (test/stream_test.lse) use a raw server that writes a response in steps the test controls. They check that:

  • a piece arrives before the rest of the body is sent;
  • the cap applies to a body by length and to one read until close;
  • the deadline stops a stalled body;
  • cancel closes the connection, as the server sees it;
  • failures before the head are errors.

The SSE parser's tests, including a property that any split of a stream parses the same, run in both back ends.