Simplify and promote using in-vm kernel

[marmarek]

The problem you're addressing (if any)
Currently by default all qubes use kernel provided by dom0. This has multiple issues:

• kernel build for dom0 environment has incompatible kernel-devel package (especially painful for Debian templates, but also an issue for different Fedora versions): kernel-devel package have broken gcc plugin #2844 kernel-devel needs tools/objtool/objtool to compile out-of-tree kernel modules #3835
• template doesn't control kernel options: enable AppArmor by default #4088
• kernel config not necessary matching template expectation (QubesOS/qubes-linux-kernel@96b8fba)

Using in-vm kernel isn't enabled by default and it's not straightforward:

• for HVM, one needs to choose "(none)" / "" kernel
• for PV, one needs to choose "pvgrub2" kernel
• for PVH, one needs to choose "pvgrub2-pvh" kernel (only recently available)

Describe the solution you'd like
Unify setting in-vm kernel - possibly translate "(none)" to "pvgrub2" if virt_mode is PV and to "pvgrub2-pvh" if virt_mode is PVH. Note that "(none)" normally is an invalid choice for PV/PVH.
This would require adjusting "(none)" label at UI level, to be less confusing/magic. Something like "(use kernel from within the qube)".
Set is as default value.

Where is the value to a user, and who might that user be?
Less deviation from template's system, according to https://www.qubes-os.org/faq/#what-is-qubes-attitude-toward-changing-guest-distros

Possible drawbacks
This change may lead also to some issues:

• less control over qube kernel means we won't be able to quickly apply qubes-specific fixes there - we'll need to wait until relevant distribution pick up updated kernel
• less control over qube kernel config - for example if some kernel feature is disabled, we may have a problem
• harder to provide extra kernel modules (fortunately we don't need u2mfn in R4.1 anymore)
• different grub2 version - for example the one in Fedora is heavily patched and grub.cfg may rely on it (for example support for "Boot Loader Specification")

Describe alternatives you've considered
Implement automatic pvgrub choice for kernel "(none)", but don't set it as default.

Relevant documentation you've consulted
https://www.qubes-os.org/doc/managing-vm-kernel/

[brendanhoar]

Peanut gallery has arrived. :)

How would Qubes dom0 be safely informed what bootloader/kernel/initrd combo is installed/available inside PV/PVH VMs? Or is there an agreed upon naming standard that is assumed?

So...Qubes already uses a notation of :object elsewhere in Qubes (e.g. devices).

Therefore, perhaps consider something along the lines of:
dom0:(default)
...
dom0:4.19.56-1.pvops
dom0:5.1.13-1.pvops
dom0:5.1.15-1.pvops
dom0:5.1.17-1.pvops
vmname:(default) [or vmname:(not specified)]
vmname:(default_for_Linux_HVM)
vmname:(default_for_Linux_PV)
vmname:(default_for_Linux_PVH)
vmname:<custom_item1_from_new_scheme>
vmname:<custom_item2_from_new_scheme>

Perhaps greying out items that aren't applicable to the current VM type?

Naming strategy allows future possibility of attempting to boot VM1 off the bootloader/kernel/initrd locally stored with VM2, if that would ever be useful (possibly just a crazy thought).

For kernel options, store per-VM changes from the default per selectable kernel, nothing if stored if the default value is left as is?

Brendan

[marmarek]

How would Qubes dom0 be safely informed what bootloader/kernel/initrd combo is installed/available inside PV/PVH VMs?

The whole idea is it wont. It will leave it to a bootloader within the VM itself, using predetermined protocol. In case of HVM, it's loading it from MBR (in the future there may be an option for UEFI). In case of PV/PVH, it's launching grub2, which will load a config from /boot within a VM.

[rustybird]

Will there still be a (non-default) Qubes provided VM kernel? I guess I should look into upstreaming 0006-block-add-no_part_scan-module-parameter.patch (used by Split dm-crypt)...

[marmarek]

Will there still be a (non-default) Qubes provided VM kernel?

I don't plan to. But there will be still (non-default) dom0-provided kernel.

[rustybird]

Will there still be a (non-default) Qubes provided VM kernel?

I don't plan to. But there will be still (non-default) dom0-provided kernel.

You mean, built by Fedora?

[marmarek]

No, same as now.

[rustybird]

Oh, I see. By "VM kernel", I meant a kernel used by the VM but located in dom0. Sorry for the confusion.

[adrelanos]

In-VM kernel all the way. Seems much more expected from all parties. (If Qubes didn't introduce dom0 kernel, I'd never expect it.) (Parties include users, developers, applications by distribution, distribution maintainers, grub.d.)

I'd go as far as making dom0 kernel user opt-in only.
If not totally deprecate dom0 kernel long term.

less control over qube kernel means we won't be able to quickly apply qubes-specific fixes there - we'll need to wait until relevant distribution pick up updated kernel

This is one of the strongest points for dom0 I have to acknowledge even though I strongly favor In-VM kernel.

For any supported template, this would be better handled by distribution specific kernel fixes inside the VM by Qubes repository?

You might say this is hard? Might be good point. But... Well, if you promote in-vm kernel, then "half" users will have one thing, and half the other thing. What you tell them then? "We have a security issue, therefore go back to dom0 provided kernel." Double maintenance effort.

[marmarek]

It's already possible to test and use in-vm kernel, even for PVH. If you want to help make it default @adrelanos , here are testing steps for PVH:

• install grub2-xen-pvh package in dom0 (from current-testing repo)
• switch kernel to pvgrub2-pvh

It should "just work" for Debian 10 and result in a working AppArmor and other features. But require more testing. In case of Whonix, which is based on Debian 10 minimal, it might require installing qubes-kernel-vm-support if not already there. And building u2mfn module (DKMS package shipped with qubes-kernel-vm-support package.

Note it won't work out of the box on Fedora 30, because of BLS mentioned in issue description. It might be enough to set GRUB_ENABLE_BLSCFG=false in /etc/default/grub and regenerate grub config.