Add usb support to io.bus

[foxtrotcz]

Hi,
this issue basically solves my previous one: #1688

I propose implementing usb-storage for use in disk via io.bus
https://qemu-project.gitlab.io/qemu/system/devices/usb.html#connecting-usb-devices

This device connects to qemu-xhci which is already implemented in Incus for USB passthrough. It also supports hotplugging.
Drivers for USB device are also preloaded in Windows installer so it can be used as media for mounting additional drivers during Windows install. It can be also used for mounting Windows install ISO itself which will then allow to install easily on NVMe disk.

I think adding this would greatly improve ability of users to install Windows without using distrobuilder.

I tested it via qemu monitor and it looks to me it works nicely.

drive_add 0 if=none,id=stick,media=cdrom,format=raw,file=/home/user/drivers.iso,file.locking=off
device_add usb-storage,bus=qemu_usb.0,drive=stick,id=usbstorage

[bensmrs]

Well, if we do this, can we do the same with network adapters?
Would solve my macOS problems :)

[foxtrotcz]

@bensmrs I will post here my chat conversation with @stgraber which basically answers this.

FoxtrotCZ
10:05 pm
@stgraber Hi, just a question about network driver in Incus. Currently there is only virtio-net. Would you consider implementing also e1000 as alternative?

I found this thread, do I understand correctly that you will not add any other alternatives or was that meant only for Snap version? First class support for alternative NIC drivers?
Thanks.

(edited)
stgraber
10:28 pm
We are trying to avoid fully virtualized drivers as much as possible as they are the main source of security issues. So as it stands, we do not support any non-paravirtualized drivers, your only way to use them is through raw.qemu which makes that instance unsupported.

FoxtrotCZ
10:30 pm
Thanks, I just ask so I dont uselessly create feature request on github that will be denied :slight_smile:

In storage you dont mind non-virtio drivers as long as they are hotpluggable.

stgraber
10:41 pm
It's the same for storage. We won't allow mtpsas and other simulated storage controllers.

virtio_scsi, virtio_blk and nvme which are our current options are fine. We also already have a USB bus, so usb_storage is fine too.

FoxtrotCZ
10:44 pm
I think now I understand it better. So basically you avoid it everywhere as much as possible but nvme and usb-storage are rare exceptions.

stgraber
11:16 pm
The way QEMU handles nvme it's effectively the same as a virtio-blk type deal, it's shared memory and basically no userspace involvement.

USB, we're less fond of, but the base machine definition in QEMU comes with a simulated USB controller that we can't remove, so the attack surface is there anyway

Though it's also a somewhat native QEMU implementation of USB, it's not faking an Intel or AMD USB controller

root@server01:~ lspci -nnn | grep -vi qemu | grep -vi virtio
00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0]
00:1f.0 ISA bridge [0601]: Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918] (rev 02)
00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922] (rev 02)
00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02)
root@server01:~
Those are the only devices that are actually emulated, none of which we can do anything about as they're all in the base machine type

11:20
but they're also reasonably simple device, with the exception of the SATA controller which I'd really like to get rid of since it's not useful at all (can't even do hotplug)

[bensmrs]

Sure, e1000 is a no-go, but what about usb-net? It’s hot-pluggable and it’s USB :)

[stgraber]

Right, the general goal on our side is to limit the number of fully emulated devices as much as possible and rely on paravirtualization wherever possible as a way to limit our attack surface.

Fully emulated devices, especially complex PCI ones are the most dangerous there.

USB is a bit more standard, at least for things like mass storage and we already have an XHCI controller anyways, so on the PCI side, there are no changes.

Looking at the USB network implementation in QEMU, it seems very lightweight on the host side, just basic ethernet passthrough, not dissimilar to the paravirtualized PCI drivers so it would probably be fine to have io.bus on NIC devices with virtio-net and usb as supported values.

[stgraber]

(Another side effect of our decision to stick to lightweight paravirtualized devices is that should we ever have a reason to move away from QEMU as the driver and switch to one of the Rust VMM based alternatives, it shouldn't be too hard as those will usually have full virtio support and USB support, but other random PCI devices aren't something that can be counted on)

[bensmrs]

Looking at the USB network implementation in QEMU, it seems very lightweight on the host side, just basic ethernet passthrough, not dissimilar to the paravirtualized PCI drivers so it would probably be fine to have io.bus on NIC devices with virtio-net and usb as supported values.

We should double-check whether mq really works on the usb-net driver, as it’s a very old implementation that I feel nobody dares touch :)
Anyway, it would really help me, so I’ll have a look after we finish the LINSTOR integration (probably with a new issue).

(Another side effect of our decision to stick to lightweight paravirtualized devices is that should we ever have a reason to move away from QEMU as the driver and switch to one of the Rust VMM based alternatives, it shouldn't be too hard as those will usually have full virtio support and USB support, but other random PCI devices aren't something that can be counted on)

… I’m already dreaming about Xen and Solo5 support :)

[bensmrs]

If no one is particularly interested in working on that, it’s only logical after #1814 that I take this one, I think.

[bensmrs]

In the meantime, you can do some scriptlet magic to achieve what you want, @foxtrotcz.
Obviously needs to be adapted, but you can take some inspiration from here -> https://discuss.linuxcontainers.org/t/incus-are-we-macos-yet/22117#p-76711-storage-remapping-9 (you need to change the device type and its bus, and probably adapt a few things here and there).

[stgraber]

I assigned it to you now, thanks!

[foxtrotcz]

@bensmrs Thanks for taking this issue. My needs (device supported by Windows installer) are met with raw.qemu setting IDE CD-ROM which is easy. I created this issue because I wanted officially supported device and usb-storage seems like the only option that works with both Windows installer and Incus hotplug requirement.

I feel bad I can only create issue and not do the work, but as I am not software developer its beyond my capabilities. So thanks again for doing this :)

[bensmrs]

My needs (device supported by Windows installer) are met with raw.qemu setting IDE CD-ROM which is easy. I created this issue because I wanted officially supported device and usb-storage seems like the only option that works with both Windows installer and Incus hotplug requirement.

To be fair, I’m not sure I completely get your use case, as I’m installing Windows with repackaged ISOs with no issue.

I feel bad I can only create issue and not do the work, but as I am not software developer its beyond my capabilities. So thanks again for doing this :)

Well you really shouldn’t; creating issues is a very important part of open source contribution. You gave me a good excuse for advancing on my macOS virtualization project, it’s also a need that I have at work so I can work on it during my office hours, and I enjoy working on the Incus codebase. It’s a win-win :)

[foxtrotcz]

To be more accurate.
I myself can and do repack Windows ISO with distrobuilder so I dont actually need this.
But I also believe there are people who just want to install Windows without any repacking and they should have the option.
Currently without repacking its just impossible (at least the supported options).
After implementing usb-storage they can then either install without Virtio on NVMe or load additional Virtio drivers from the USB during install and use Virtio-SCSI.

For example person who is used to Proxmox will be able to transition to Incus better if they dont need to repack.

So my "use case" is that I want Incus to be more accessible to regular users.

Plus, recently Truenas adopted Incus so I bet more regular users will start using Incus.