Preliminary support for AWS Nitro Enclaves

[tylerfanelli]

This PR adds a new flavor, libkrun-nitro. The nitro flavor is expected to be run on AWS EC2 instances that support AWS Nitro Enclaves.

AWS Nitro Enclaves are fully isolated and highly constrained virtual machines that run on the Nitro hypervisor. CPUs and memory of an enclave are completely isolated from a parent instance and other users of a system, significantly reducing the attack surface and helping to prevent the stealing/tampering of data.

Try for yourself

To demo on an AWS EC2 instance with Nitro Enclaves enabled:

• Clone repo, checkout the nitro branch

$ git clone https://github.com/tylerfanelli/libkrun.git
$ cd libkrun
$ git checkout nitro

• Build the libkrun-nitro flavor

$ make NITRO=1
$ make install NITRO=1
$ ln /usr/local/lib64/libkrun-nitro.so /usr/local/lib64/libkrun.so.1

• Build the nitro example

$ cd examples
$ make nitro

• Run the nitro example with an example EIF file found in examples/test_data

$ LD_LIBRARY_PATH=/usr/local/lib64 ./nitro test_data/hello.eif

This will run the enclave in debug mode, where you can examine debug output from the nitro enclave.

This will include Linux kernel logs:

[    0.000000] Booting Linux on physical CPU 0x0                                                                                                                                                                                                                                
.. snip ..

As well as a hello message from the enclave:

.. snip ..
[    0.062913] random: crng init done
[    0.063151] NSM RNG: returning rand bytes = 128
[    0.063450] NSM RNG: returning rand bytes = 128
Hello (from a nitro enclave!)
[    0.066406] reboot: Restarting system

[tylerfanelli]

cc @slp

[jakecorrenti]

Great work! I haven't tested the code yet, but so far things are looking good. I added a couple nits here and there.

In general, I would prefer if the commits were ordered such that changes in src/ come first, then include/, then finally examples/. My thought process behind this is it prevents the use of the API before the functionality is fully baked. Then, once we have the "backend" code, we can add the UAPI, then finally the example when everything's in a state where a working solution is possible. I think that would be a pretty invasive change here so I'm not asking you to do that, but I'm not sure how others feel about commit ordering in that sense.

[tylerfanelli]

In general, I would prefer if the commits were ordered such that changes in src/ come first, then include/, then finally examples/. My thought process behind this is it prevents the use of the API before the functionality is fully baked. Then, once we have the "backend" code, we can add the UAPI, then finally the example when everything's in a state where a working solution is possible. I think that would be a pretty invasive change here so I'm not asking you to do that, but I'm not sure how others feel about commit ordering in that sense.

I would say this is up to personal preference. You could include the examples/ and include/ modifications along with the src/ modifications if it logically makes sense, or you could split them up.

However, this is not true if you make a change that would break a caller's use of an API. In that scenario, the changes to the caller must be also included (in examples/, for instance), as to not break the example/test.

[jakecorrenti]

question: do we want to follow some sort of naming convention with the TEE features? We currently have amd-sev which is prefixed by the manufacturer, intel-tdx which will follow that style as well. We also have nitro and in the future cca.

Should we come to some sort of consensus with regards to a manufacturer prefix or not? Does it matter?

[tylerfanelli]

We could. I prefer sev and tdx rather than amd-sev and intel-tdx FWIW.

[jakecorrenti]

Code LGTM

[tylerfanelli]

@slp Can you review when you have the chance?

[tylerfanelli]

@jakecorrenti @slp After rebasing, I'm seeing a weird issue. Each test is failing with the following:

error[E0308]: mismatched types
   --> src/devices/src/legacy/ioapic.rs:141:28
    |
141 |         vm.set_gsi_routing(&irq_routing[0])?;
    |            --------------- ^^^^^^^^^^^^^^^ expected `&FamStructWrapper<kvm_irq_routing>`, found `&kvm_irq_routing`
    |            |
    |            arguments to this method are incorrect
    |
    = note: expected reference `&vmm_sys_util::fam::FamStructWrapper<kvm_bindings::kvm_irq_routing>`
               found reference `&kvm_bindings::kvm_irq_routing`
note: method defined here
   --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/kvm-ioctls-0.22.0/src/ioctls/vm.rs:633:12
    |
633 |     pub fn set_gsi_routing(&self, irq_routing: &KvmIrqRouting) -> Result<()> {
    |            ^^^^^^^^^^^^^^^

I didn't change any of the kvm-bindings, kvm-ioctls, or vmm-sys-utils dependencies, and this is not an issue in the main branch (with the same dependency versions). Do you have any idea what could cause this?

[jakecorrenti]

@jakecorrenti @slp After rebasing, I'm seeing a weird issue. Each test is failing with the following:

error[E0308]: mismatched types
   --> src/devices/src/legacy/ioapic.rs:141:28
    |
141 |         vm.set_gsi_routing(&irq_routing[0])?;
    |            --------------- ^^^^^^^^^^^^^^^ expected `&FamStructWrapper<kvm_irq_routing>`, found `&kvm_irq_routing`
    |            |
    |            arguments to this method are incorrect
    |
    = note: expected reference `&vmm_sys_util::fam::FamStructWrapper<kvm_bindings::kvm_irq_routing>`
               found reference `&kvm_bindings::kvm_irq_routing`
note: method defined here
   --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/kvm-ioctls-0.22.0/src/ioctls/vm.rs:633:12
    |
633 |     pub fn set_gsi_routing(&self, irq_routing: &KvmIrqRouting) -> Result<()> {
    |            ^^^^^^^^^^^^^^^

I didn't change any of the kvm-bindings, kvm-ioctls, or vmm-sys-utils dependencies, and this is not an issue in the main branch (with the same dependency versions). Do you have any idea what could cause this?

upstream kvm-bindings wrapped kvm_irq_routing in a FamStructWrapper as of this PR rust-vmm/kvm#325. I can work on a fix right now

[tylerfanelli]

Yes, but I never updated those dependencies? Regardless, it seems that #348 fixed it. Thanks!

[jakecorrenti]

Yes, but I never updated those dependencies? Regardless, it seems that #348 fixed it. Thanks!

You changed src/utils/Cargo.toml to use kvm-bindings >= 0.10 instead of 0.11 which probably triggered the update if I had to guess

[tylerfanelli]

Yes, but I never updated those dependencies? Regardless, it seems that #348 fixed it. Thanks!

You changed src/utils/Cargo.toml to use kvm-bindings >= 0.10 instead of 0.11 which probably triggered the update if I had to guess

Yea, that must've been a mistake. Removed that change.

[slp]

LGTM