Out of memory #73
Comments
|
Trying again with a stripped unikernel image (9.1 MB) it OOM'd in a different place: |
|
Thanks for your report. My first observation: something catches out of memory. I guess for a web server that by discarding one (large) request can free up a good chunk of memory, this is fine. Although I'm not entirely convinced. From a second observation: when we receive the unikernel image, we have it in the request in memory, then do the multipart decoding (keeping it in memory a second time)? Can we revise/dig more into the details to not need it multiple times in memory -- eventually even streaming the binary from the request to albatross (I guess this will be pretty hard, since we'd need a streaming x509 certificate signing request / signing API). Instead of taking the request in full, then multipart decoding in full, then creating the certificate signing request & certificate -- why not stream the request and stream the decoding (or do an in-place decoding without the need of more memory)? |
|
Further discussion lead to:
Best achieved likely with seq. Requires changes to asn1-combinators. But there'll be a benefit in other packages (such as albatross). But this will take some time to push through. |
|
As alternative for the out path, we could modify the interface albatross has (and e.g. allow the unikernel binary on the TLS channel, instead of in the client certificate). |
|
/cc @kit-ty-kate here So, from what I can tell there's many things that go wrong here ;) One is that we should have a streaming interface. The other is ocaml-tls going a bit nuts when you say "please send this file". How it works is that let encrypt_records encryptor version records =
let rec split = function
| [] -> []
| (t1, a) :: xs when String.length a >= 1 lsl 14 ->
let fst, snd = split_str a (1 lsl 14) in
(t1, fst) :: split ((t1, snd) :: xs)
| x::xs -> x :: split xs
and crypt st = function
| [] -> (st, [])
| (ty, buf)::rs ->
let (st, ty, enc) = encrypt version st ty buf in
let (st, encs) = crypt st rs in
(st, (ty, enc) :: encs)
in
crypt encryptor (split records)
let split_str ?(start = 0) str off =
String.sub str start off,
String.sub str (start + off) (String.length str - off - start)Now, the TLS 1.3 AEAD encryption works by adding the content type at the end - another allocate -- and another one by let encrypt (version : tls_version) (st : crypto_state) ty buf =
match st with
| None -> (st, ty, buf)
| Some ctx ->
match version with
| `TLS_1_3 ->
(match ctx.cipher_st with
| AEAD c ->
let buf =
let t = String.make 1 (Char.unsafe_chr (Packet.content_type_to_int ty)) in
buf ^ t
in
let nonce = Crypto.aead_nonce c.nonce ctx.sequence in
let adata = Crypto.adata_1_3 (String.length buf + Crypto.tag_len c.cipher) in
let buf = Crypto.encrypt_aead ~cipher:c.cipher ~adata ~key:c.cipher_secret ~nonce buf in
(Some { ctx with sequence = Int64.succ ctx.sequence }, Packet.APPLICATION_DATA, buf)
| _ -> assert false)So, we first split it into 2^14 chunks, allocating quite a lot of memory, just to allocate these once more for the additional content type byte, and then allocate these once more for the encrypted data. Consider a chunk of 131073 bytes (2 * 2^14 + 1), we split it into 3 chunks - allocating a 65536, one 65537; one 65536 and one 1 string, then allocating 65537, 65537, 2; and once more 65537, 65537, 2 bytes --> blowing up by around a factor of 5. Plus likely due to MirageOS network stack still using Cstruct some further buffers... TL;DR: we need to fix allocations and re-evaluate split_str usage, as well as |
|
since we improved ocaml-tls (release 2.0.1), I'll close this issue -- the currently running mollymawk allows to deploy e.g. unipi :) |
I triggered an out of memory exception while trying to deploy a ~17 MB unikernel with no arguments. The mollymawk unikernel itself runs with "only" 128 MB.
The text was updated successfully, but these errors were encountered: