VM launch fails due to an AppArmor rule violation

[dshcherb]

Is there an existing issue for this?

• There is no existing issue for this bug

Is this happening on an up to date version of Incus?

• This is happening on a supported version of Incus

Incus system details

➜  ~ incus info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- images_all_projects
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
- disk_io_bus_virtio_blk
- loki_config_instance
- instance_create_start
- clustering_evacuation_stop_options
- boot_host_shutdown_action
- agent_config_drive
- network_state_ovn_lr
- image_template_permissions
- storage_bucket_backup
- storage_lvm_cluster
- shared_custom_block_volumes
- auth_tls_jwt
- oidc_claim
- device_usb_serial
- numa_cpu_balanced
- image_restriction_nesting
- network_integrations
- instance_memory_swap_bytes
- network_bridge_external_create
- network_zones_all_projects
- storage_zfs_vdev
- container_migration_stateful
- profiles_all_projects
- instances_scriptlet_get_instances
- instances_scriptlet_get_cluster_members
- instances_scriptlet_get_project
- network_acl_stateless
- instance_state_started_at
- networks_all_projects
- network_acls_all_projects
- storage_buckets_all_projects
- resources_load
- instance_access
- project_access
- projects_force_delete
- resources_cpu_flags
- disk_io_bus_cache_filesystem
- instance_oci
- clustering_groups_config
- instances_lxcfs_per_instance
- clustering_groups_vm_cpu_definition
- disk_volume_subpath
- projects_limits_disk_pool
- network_ovn_isolated
- qemu_raw_qmp
- network_load_balancer_health_check
- oidc_scopes
- network_integrations_peer_name
- qemu_scriptlet
- instance_auto_restart
- storage_lvm_metadatasize
- ovn_nic_promiscuous
- ovn_nic_ip_address_none
- instances_state_os_info
- network_load_balancer_state
- instance_nic_macvlan_mode
- storage_lvm_cluster_create
- network_ovn_external_interfaces
- instances_scriptlet_get_instances_count
- cluster_rebalance
- custom_volume_refresh_exclude_older_snapshots
- storage_initial_owner
- storage_live_migration
- instance_console_screenshot
- image_import_alias
- authorization_scriptlet
- console_force
- network_ovn_state_addresses
- network_bridge_acl_devices
- instance_debug_memory
- init_preseed_storage_volumes
- init_preseed_profile_project
- instance_nic_routed_host_address
- instance_smbios11
- api_filtering_extended
- acme_dns01
- security_iommu
- network_ipv4_dhcp_routes
- network_state_ovn_ls
- network_dns_nameservers
- acme_http01_port
- network_ovn_ipv4_dhcp_expiry
- instance_state_cpu_time
- network_io_bus
- disk_io_bus_usb
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: dima
auth_user_method: unix
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    <redacted>
  certificate_fingerprint: <redacted>
  driver: lxc | qemu
  driver_version: 6.0.3 | 9.2.2
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.8.0-55-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "24.04"
  project: default
  server: incus
  server_clustered: false
  server_event_mode: full-mesh
  server_name: ws
  server_pid: 260356
  server_version: 6.10.1
  storage: dir
  storage_version: "1"
  storage_supported_drivers:
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0
    remote: false
  - name: lvmcluster
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0
    remote: true

Instance details

➜  ~ incus config show bgp-agent                              
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu noble amd64 (20250325_07:42)
  image.os: Ubuntu
  image.release: noble
  image.requirements.cgroup: v2
  image.serial: "20250325_07:42"
  image.type: disk-kvm.img
  image.variant: default
  volatile.base_image: b908e3c6e28e401e0af5fe4bbbf79468dedc6c59b7becf8a217ab783ce0e60e9
  volatile.cloud-init.instance-id: 9d3ba1c4-6883-4bb5-88d0-dd454b35ec54
  volatile.eth0.hwaddr: <redacted>
  volatile.uuid: 55969559-af3b-409c-9563-81024aebdb75
  volatile.uuid.generation: 55969559-af3b-409c-9563-81024aebdb75
  volatile.vsock_id: "2038216636"
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

Instance log

➜  ~ incus info --show-log bgp-agent
Name: bgp-agent
Description: 
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2025/03/26 03:15 MSK
Last Used: 1970/01/01 03:00 MSK

Log:

qemu-system-x86_64:/run/incus/bgp-agent/qemu.conf:46: Could not open '/usr/share/OVMF/OVMF_CODE_4M.ms.fd': Permission denied

Current behavior

➜  ~ incus launch --vm images:ubuntu/24.04 bgp-agent
Launching bgp-agent
Error: Failed instance creation: Failed to run: forklimits limit=memlock:unlimited:unlimited fd=3 fd=4 -- /opt/incus/bin/qemu-system-x86_64 -S -name bgp-agent -uuid 55969559-af3b-409c-9563-81024aebdb75 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /run/incus/bgp-agent/qemu.conf -spice unix=on,disable-ticketing=on,addr=/run/incus/bgp-agent/qemu.spice -pidfile /run/incus/bgp-agent/qemu.pid -D /var/log/incus/bgp-agent/qemu.log -smbios type=2,manufacturer=LinuxContainers,product=Incus -runas incus: qemu-system-x86_64: -runas incus: warning: -runas is deprecated, use '-run-with user=...' instead
: exit status 1

➜  ~ journalctl -k --no-pager --since '1m ago' | grep apparmor
мар 26 03:14:56 ws kernel: audit: type=1400 audit(1742948096.540:324): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-bgp-agent_</var/lib/incus>" pid=269471 comm="apparmor_parser"
мар 26 03:15:16 ws kernel: audit: type=1400 audit(1742948116.119:325): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus_archive-a70a73fc-6fbf-4de3-9c39-d769efb177c2" pid=269502 comm="apparmor_parser"
мар 26 03:15:16 ws kernel: audit: type=1400 audit(1742948116.146:326): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_archive-a70a73fc-6fbf-4de3-9c39-d769efb177c2" pid=269507 comm="apparmor_parser"
мар 26 03:15:16 ws kernel: audit: type=1400 audit(1742948116.163:327): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus_qemu-img-<var-lib-incus-images-b908e3c6e28e401e0af5fe4bbbf79468dedc6c59b7becf8a217ab783ce0e60e9.rootfs>_<var-lib-incus-storage-pools-default-virtual-machines-bgp-agent-root.img>" pid=269509 comm="apparmor_parser"
мар 26 03:15:16 ws kernel: audit: type=1400 audit(1742948116.175:328): apparmor="DENIED" operation="open" class="file" profile="incus_qemu-img-<var-lib-incus-images-b908e3c6e28e401e0af5fe4bbbf79468dedc6c59b7becf8a217ab783ce0e60e9.rootfs>_<var-lib-incus-storage-pools-default-virtual-machines-bgp-agent-root.img>" name="/sys/devices/system/node/" pid=269510 comm="qemu-img" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
мар 26 03:15:16 ws kernel: audit: type=1400 audit(1742948116.192:329): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_qemu-img-<var-lib-incus-images-b908e3c6e28e401e0af5fe4bbbf79468dedc6c59b7becf8a217ab783ce0e60e9.rootfs>_<var-lib-incus-storage-pools-default-virtual-machines-bgp-agent-root.img>" pid=269514 comm="apparmor_parser"
мар 26 03:15:16 ws kernel: audit: type=1400 audit(1742948116.209:330): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus_qemu-img-<var-lib-incus-images-b908e3c6e28e401e0af5fe4bbbf79468dedc6c59b7becf8a217ab783ce0e60e9.rootfs>_<var-lib-incus-storage-pools-default-virtual-machines-bgp-agent-root.img>" pid=269516 comm="apparmor_parser"
мар 26 03:15:16 ws kernel: audit: type=1400 audit(1742948116.220:331): apparmor="DENIED" operation="open" class="file" profile="incus_qemu-img-<var-lib-incus-images-b908e3c6e28e401e0af5fe4bbbf79468dedc6c59b7becf8a217ab783ce0e60e9.rootfs>_<var-lib-incus-storage-pools-default-virtual-machines-bgp-agent-root.img>" name="/sys/devices/system/node/" pid=269517 comm="qemu-img" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
мар 26 03:15:17 ws kernel: audit: type=1400 audit(1742948117.178:332): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus_qemu-img-<var-lib-incus-images-b908e3c6e28e401e0af5fe4bbbf79468dedc6c59b7becf8a217ab783ce0e60e9.rootfs>_<var-lib-incus-storage-pools-default-virtual-machines-bgp-agent-root.img>" pid=269529 comm="apparmor_parser"
мар 26 03:15:18 ws kernel: audit: type=1400 audit(1742948118.288:333): apparmor="STATUS" operation="profile_load" profile="unconfined" name="incus-bgp-agent_</var/lib/incus>" pid=269550 comm="apparmor_parser"
мар 26 03:15:18 ws kernel: audit: type=1400 audit(1742948118.328:334): apparmor="DENIED" operation="open" class="file" profile="incus-bgp-agent_</var/lib/incus>" name="/usr/share/OVMF/OVMF_CODE_4M.secboot.fd" pid=269565 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

➜  ~ ls -al /usr/share/OVMF         
total 8744
drwxr-xr-x   2 root root    4096 дек 19 16:56 .
drwxr-xr-x 427 root root   20480 фев 23 16:03 ..
-rw-r--r--   1 root root 3653632 окт 25 20:25 OVMF_CODE_4M.fd
lrwxrwxrwx   1 root root      23 окт 25 20:25 OVMF_CODE_4M.ms.fd -> OVMF_CODE_4M.secboot.fd
-rw-r--r--   1 root root 3653632 окт 25 20:25 OVMF_CODE_4M.secboot.fd
lrwxrwxrwx   1 root root      23 окт 25 20:25 OVMF_CODE_4M.snakeoil.fd -> OVMF_CODE_4M.secboot.fd
-rw-r--r--   1 root root  540672 окт 25 20:25 OVMF_VARS_4M.fd
-rw-r--r--   1 root root  540672 окт 25 20:25 OVMF_VARS_4M.ms.fd
-rw-r--r--   1 root root  540672 окт 25 20:25 OVMF_VARS_4M.snakeoil.fd

Checked the SHA256 on a working system and on a broken system - both are the same as below:

➜  ~ sha256sum /usr/share/OVMF/OVMF_CODE_4M.ms.fd                
9e7cb2ee2fd25f7ec5c825dc0b85d924249e9aad3380d00321151268885dcff3  /usr/share/OVMF/OVMF_CODE_4M.ms.fd

Expected behavior

A profile gets generated such that a VM can start.

Steps to reproduce

On the system where it happens it is enough to run incus launch --vm images:ubuntu/24.04 <vm-name> to reproduce.

Trying to find a useful diff with a working system to understand why this started happening.

[dshcherb]

# ...
[pid 272837] unlink("/run/incus/bgp-agent/qemu.monitor") = 0
[pid 272837] bind(15, {sa_family=AF_UNIX, sun_path="/run/incus/bgp-agent/qemu.monitor"}, 110) = 0
[pid 272837] listen(15, 1)              = 0
[pid 272837] getpeername(15, 0x5fdbf2f13f80, [128]) = -1 ENOTCONN (Transport endpoint is not connected)
[pid 272837] getsockname(15, {sa_family=AF_UNIX, sun_path="/run/incus/bgp-agent/qemu.monitor"}, [128 => 36]) = 0
[pid 272837] getsockname(15, {sa_family=AF_UNIX, sun_path="/run/incus/bgp-agent/qemu.monitor"}, [128 => 36]) = 0
[pid 272837] write(10, "\1\0\0\0\0\0\0\0", 8) = 8
[pid 272837] mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76329ae86000
[pid 272837] openat(AT_FDCWD, "/usr/share/OVMF/OVMF_CODE_4M.ms.fd", O_RDONLY|O_NONBLOCK|O_CLOEXEC) = -1 EACCES (Permission denied)
[pid 272837] newfstatat(AT_FDCWD, "/usr/share/OVMF/OVMF_CODE_4M.ms.fd", {st_mode=S_IFREG|0644, st_size=3653632, ...}, 0) = 0
[pid 272837] openat(AT_FDCWD, "/dev/urandom", O_RDONLY|O_CLOEXEC) = 16
[pid 272837] read(16, "\203\333v\232w\237V[&\316b\27\224$x\322", 16) = 16
[pid 272837] close(16)                  = 0
[pid 272837] futex(0x76329f06fde8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid 272837] openat(AT_FDCWD, "/dev/null", O_RDWR) = 16
[pid 272837] fcntl(16, F_OFD_GETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0, l_pid=0}) = 0
[pid 272837] close(16)                  = 0
[pid 272837] openat(AT_FDCWD, "/usr/share/OVMF/OVMF_CODE_4M.ms.fd", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
# ...

[dshcherb]

The instance profile that got generated:

root@ws:/var/lib/incus/security/apparmor/profiles# cat incus-bgp-agent 
#include <tunables/global>
profile "incus-bgp-agent_</var/lib/incus>" flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/base>
  #include <abstractions/consoles>
  #include <abstractions/nameservice>

  capability dac_override,
  capability dac_read_search,
  capability ipc_lock,
  capability setgid,
  capability setuid,
  capability sys_chroot,
  capability sys_rawio,
  capability sys_resource,

  # Needed by qemu
  /dev/hugepages/**                         rw,
  /dev/kvm                                  rw,
  /dev/net/tun                              rw,
  /dev/ptmx                                 rw,
  /dev/sev                                  rw,
  /dev/vfio/**                              rw,
  /dev/vhost-net                            rw,
  /dev/vhost-vsock                          rw,
  /etc/machine-id                           r,
  /run/udev/data/*                          r,
  @{PROC}/sys/vm/max_map_count              r,
  @{PROC}/@{pid}/cpuset                     r,
  @{PROC}/@{pid}/task/*/comm                rw,
  /sys/bus/                                 r,
  /sys/bus/nd/devices/                      r,
  /sys/bus/usb/devices/                     r,
  /sys/bus/usb/devices/**                   r,
  /sys/class/                               r,
  /sys/devices/**                           r,
  /sys/module/vhost/**                      r,
  /tmp/incus_sev_*                          r,
  /opt/incus/share/qemu/**                         kr,
  /usr/share/qemu/**                        kr,
  /usr/share/seabios/**                     kr,
  /etc/nsswitch.conf         r,
  /etc/passwd                r,
  /etc/group                 r,
  @{PROC}/version                           r,

  # Extra config paths


  # Extra binaries
  /opt/incus/bin/qemu-system-x86_64                            mrix,

  # Used by qemu for live migration NBD server and client
  unix (bind, listen, accept, send, receive, connect) type=stream,

  # Instance specific paths
  /var/log/incus/bgp-agent/** rwk,
  /run/incus/bgp-agent/** rwk,
  /var/lib/incus/storage-pools/default/virtual-machines/bgp-agent/** rwk,
  /var/lib/incus/devices/bgp-agent/** rwk,

  # Needed for the fork sub-commands
  /opt/incus/bin/incusd mr,
  @{PROC}/@{pid}/cmdline r,
  /{etc,lib,usr/lib}/os-release r,

  # Things that we definitely don't need
  deny @{PROC}/@{pid}/cgroup r,
  deny /sys/module/apparmor/parameters/enabled r,
  deny /sys/kernel/mm/transparent_hugepage/hpage_pmd_size r,
  deny /etc/gss/mech.d/ r,
  deny /etc/ssl/openssl.cnf r,

/opt/incus/agent/ r,
  /opt/incus/agent/* r,

# Entries from LD_LIBRARY_PATH

  /opt/incus/lib//** mr,
}

[dshcherb]

Looks like I found the diff:

# Broken system
sudo grep OVMF /run/incus/bgp-agent/qemu.conf 
file = "/usr/share/OVMF/OVMF_CODE_4M.ms.fd"

# Working system:
sudo grep OVMF /run/incus/bgp-agent/qemu.conf
file = "/opt/incus/share/qemu/OVMF_CODE.4MB.fd"

Not sure why it points to the /usr/share/OVMF instead of /opt/incus/share/qemu.

[dshcherb]

The INCUS_EDK2_PATH envar seems to be set correctly on the "broken" system and matches the working one:

sudo xargs -0 -L1 -a  /proc/$(pgrep -f incusd)/environ | grep EDK
INCUS_EDK2_PATH=/opt/incus/share/qemu/

[dshcherb]

GetArchitectureFirmwarePairsForUsage (https://github.com/lxc/incus/blob/v6.10.1/internal/server/instance/drivers/driver_qemu.go#L2173-L2177) should normally pick up the INCUS_EDK2_PATH environment variable ( https://github.com/lxc/incus/blob/v6.10.1/internal/server/instance/drivers/edk2/driver_edk2.go#L179-L205). Looks like it doesn't in my case.

[stgraber]

I suspect one system has a pre-existing OVMF installation and the other one doesn't, but yeah, that suggests a regression on our end as the environment variable should take precedence on any locally available data.

[stgraber]

Reproduced the issue here

[dshcherb]

Verified with the latest build after recreating the VM:

➜  ~ sudo grep OVMF /run/incus/bgp-agent/qemu.conf
file = "/opt/incus/share/qemu/OVMF_CODE.4MB.fd"

➜  ~ sudo apt policy incus 
incus:
  Installed: 1:0~ubuntu24.04~202503281128
  Candidate: 1:0~ubuntu24.04~202503281128
  Version table:
 *** 1:0~ubuntu24.04~202503281128 500
        500 https://pkgs.zabbly.com/incus/daily noble/main amd64 Packages
        100 /var/lib/dpkg/status

Thanks a lot!