fetch
An HTTP/1.1 client in pure Louise for fetching what a URL points to: one
GET, over TCP or TLS, guarded against server-side request forgery (SSRF).
Built for link previews; depends on packages/url (ADR 0079).
Needs engineering review before production use: it is an SSRF surface. ADR 0079 lists the open questions.
import fetch.{Options, Response}
use config <- result.try (tls.config (tls.defaults ())) // once, and share it
use page <- result.try (url.parse "https://example.com/post")
let opts = Options(..fetch.options config, max_body: 2097152)
match fetch.get page [("User-Agent", "toki/1")] opts
{ Ok(Response(200, headers, body, truncated)) -> ...
, Ok(Response(status, ..)) -> ... // 3xx included: redirects are not followed
, Error(e) -> ...
}
API
pub type Policy { PublicOnly | AnyAddress }
pub type Options { Options(tls: Resource<TlsConfig>, policy: Policy,
max_body: Int, max_head: Int, deadline: Int) }
pub fn options :: config: Resource<TlsConfig> -> Options / <clock>
pub fn deadline_in :: ms: Int -> Int / <clock>
pub type Response { Response(status: Int, headers: List<(String, String)>,
body: Bytes, truncated: Bool) }
pub fn header :: resp: Response name: String -> Option<String>
pub type Error
{ BadUrl(String) | BadHeader(String) | Refused(host: String, ip: String)
| NoAddress(String) | Connect(String) | Tls(TlsError) | Malformed(String) | TimedOut }
pub fn get :: target: Url headers: List<(String, String)> opts: Options
-> Result<Response, Error> / <proc, clock>
pub fn request :: method: String target: Url headers: List<(String, String)>
body: Bytes opts: Options -> Result<Response, Error> / <proc, clock>
requestisgetwith any method and a body (ADR 0079, Revision 1):fetch.request "POST" api [("Content-Type", "application/json")] body opts. The body is sent with itsContent-Length(a POST or PUT without one says0); the caller's ownContent-Lengthis dropped with the other framing headers. The method must be a token.getisrequest "GET" … (bytes.empty ()).options configisPublicOnly, a 1 MiB body, a 64 KiB head and a deadline 10 seconds from now. Change it with a record update.deadlineis an instant oftime.monotonic, in milliseconds:deadline_in 10000. One deadline can cover a whole redirect chain.headerignores case and gives the first value.
What get does
- Checks before any connection. Only
httpandhttps. The URL is parsed again from its text, so a hand-builtUrlgets the same checks as a parsed one. A header name must be a token and a value must not hold a CR, LF or NUL (BadHeader). - Resolves once, checks every address.
net.resolvegives the host's addresses. UnderPublicOnly, if any of them is not public (fetch/address), the fetch isRefused(host, ip). Then it connects to those same addresses, in order, until one accepts: the name is not resolved again, so a second lookup cannot answer differently (DNS rebinding). - TLS for
https, with the caller's configuration. The host name is the server name sent (SNI) and the name the certificate must have. - Sends
GET <path?query> HTTP/1.1,Host(with the port when it is not the scheme's),Connection: close,Accept-Encoding: identity(there is no decompression), then the caller's headers. The caller'sHost,Connection,Accept-Encoding,Content-Length,Transfer-Encoding,TEandUpgradeare dropped. - Reads the head (at most
max_headbytes), skipping1xxresponses, then the body byContent-Length,chunked, or until the server closes. AnotherTransfer-Encoding, disagreeingContent-Lengths, or a connection that ends before the body does areMalformed. A TLS stream that ends withoutclose_notifyends a body read until close. - Keeps at most
max_bodybytes and stops reading there, withtruncated: True. - Never follows redirects. A 3xx is returned as it is. To follow one,
resolve
Locationwithurl.resolveand callgetagain with the same deadline: the new host is checked like the first. - Deadline. The exchange runs in an actor of its own. At the
deadline the caller kills it and answers
TimedOut, wherever it was waiting. A kill that lands while the actor waits innet.resolve(on theioblocking pool) takes effect when the lookup returns; the caller does not wait for that.
Streamed responses
For server-sent events and other long answers, request_streaming returns
once the response head has arrived. You then read the body as it comes
(ADR 0079, Revision 2):
pub type Streamed { Streamed(status: Int, headers: List<(String, String)>, body: BodyStream) }
pub type Piece { Data(Bytes) | End(truncated: Bool) }
pub fn request_streaming :: method: String target: Url headers: List<(String, String)>
body: Bytes opts: Options -> Result<Streamed, Error> / <proc, clock>
pub fn next :: stream: BodyStream -> Result<Piece, Error> / <proc, clock>
pub fn cancel :: stream: BodyStream -> () / <proc>
- Same checks as
request: the URL, the headers, the address policy and the limits. A failure before the head is theErrorofrequest_streaming. - Reading:
nextgives the body's bytes as they arrive (a chunked body already decoded), thenEnd. AfterEndor anErrorthe connection is closed. - Deadline:
opts.deadlinecovers the whole exchange, body included, so give a long answer a long deadline. At the deadlinenextanswersTimedOutand stops the exchange. - Cancelling:
cancelcloses the connection, so the server sees the client leave, and drops what arrived unread. Call it when you stop reading early. It is harmless after the end. - Memory: at most
max_bodybytes are delivered, so a reader that falls behind holds at most that much in its mailbox. - Ownership: only the actor that made the request can read the stream.
fetch/sse parses server-sent events (text/event-stream) from those
pieces:
import sse
let (parser, events) = sse.feed parser piece // [Event(name: "delta", data: "Bon", id: "")]
It follows the HTML standard's rules:
- lines end with CRLF, LF or CR, even when one is split across pieces;
- a blank line dispatches an event that has data;
event,dataandidare read, and theidpersists from one event to the next;retryis readable withsse.retry;- comments and unknown fields are ignored;
- a leading byte order mark is dropped.
Reconnecting is the caller's choice.
fetch/address
classify ip -> Result<Reach, String> with Reach { Public | Private(reason) },
and is_public ip. Refused ranges:
| IPv4 | IPv6 |
|---|---|
0.0.0.0/8 this network | :: unspecified, ::1 loopback |
10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 private | fc00::/7 unique local |
100.64.0.0/10 carrier-grade NAT | fe80::/10 link-local, fec0::/10 site-local |
127.0.0.0/8 loopback | ff00::/8 multicast |
169.254.0.0/16 link-local (cloud metadata) | ::/96 IPv4-compatible |
224.0.0.0 and up: multicast, reserved, broadcast |
::ffff:a.b.c.d (IPv4-mapped) and 64:ff9b::/96 (NAT64) are judged by
the IPv4 address they hold. Only the forms net.resolve gives are read:
anything else (leading zeros, a zone, a host name) is an Error, and
PublicOnly refuses it.
fetch/response
The pure parsers get uses, tested on their own: head data max_head
(Ok(None) while incomplete), framing head, and a chunked decoder fed
as bytes arrive (dechunker max, feed, is_done, body, and take,
which hands out what was decoded so far for a streamed body).
Tests
The parsers have property tests in both back ends. The loopback tests
(std/http, a raw TCP server, a TLS server with the fixtures of
test/fixtures/tls) need sockets, which only native code
has, so they run when LOUISE_NET_TESTS is set:
LOUISE_NET_TESTS=1 louise test --native
The streamed responses' tests (test/stream_test.lse) use a raw server that writes a response in steps the test controls. They check that:
- a piece arrives before the rest of the body is sent;
- the cap applies to a body by length and to one read until close;
- the deadline stops a stalled body;
cancelcloses the connection, as the server sees it;- failures before the head are errors.
The SSE parser's tests, including a property that any split of a stream parses the same, run in both back ends.