Unable to deploy initial zone/setup - Fails at creating guest network

[sjanssen15]

problem

We are trying to setup CloudStack on a single DL380 and setup the initial first zone (and eventually add more KVM hosts that are ready). When running through the setup we find the following error:
Creating guest Network | Something went wrong; please correct the following: Provider VirtualRouter is either not enabled or doesn't support service Dhcp in physical network id=211

I tried to click on "Fix errors" but it brings me back to storage networks tab. After this, the setup keeps looping on the setup.

I had a look at the management-server.log file but don't see anything jump out.

Some context:
Host with 2 bridges configured to physical adapters:

• cloudbr0 = management & storage traffic on VLAN 1451
• cloudbr1 = guest on VXLAN VNI 5000 (also tried VLAN isolation for guest, same error)

Storage:

• Local NFS server at /export/primary and /export/secondary

Log file

See line 30819 for instance
management-server.log

versions

ACS: 4.20.0
OS: Ubuntu 24.04.2 LTS
Kernel: 6.8.0-57-generic
libvirtd: 10.0.0, package 10.0.0-2ubuntu8.6
cloudstack-agent: noble/4.20.0.0
Architecture: just 1 host for now to initial setup, bare metal HP DL380 gen11
NIC: 1 adapter with 4 nics
Storage: For now local NFS server on the same host

The steps to reproduce the bug

1. Install Ubuntu 24.04 02 LTS and configure by default management install guide for 4.20 and hypervisor guide
2. Get UI working and start the initial setup for the first zone

What to do about it?

No response

[Pearl1594]

Can you please share the output of the following command, using cmk:

cmk list networkserviceproviders physicalnetworkid=211 name=VirtualRouter

Just to validate if it is enabled and all the services are present.

[sjanssen15]

Thanks for taking time to reply. The output

(cbs) 🐱 > list networkserviceproviders physicalnetworkid=211 name=VirtualRouter
{
  "count": 1,
  "networkserviceprovider": [
    {
      "canenableindividualservice": true,
      "id": "9282a0af-8598-4daa-a746-55465ca23745",
      "name": "VirtualRouter",
      "physicalnetworkid": "44f8f8fa-3025-441e-8a76-086df49dbff3",
      "servicelist": [
        "Vpn",
        "Dhcp",
        "Dns",
        "Gateway",
        "Firewall",
        "Lb",
        "SourceNat",
        "StaticNat",
        "PortForwarding",
        "UserData"
      ],
      "state": "Enabled"
    }
  ]
}

[Pearl1594]

Can you also check if the virtual router element for the given nspid is enabled.

cmk list virtualrouterelements nspid=9282a0af-8598-4daa-a746-55465ca23745

If enabled: false then can you please run the following:

cmk configure virtualrouterelement id=<virtual_router_element_uuid_from_above_cmd> enabled=true

This last step can be done on the ui as well

[sjanssen15]

I do see however that the system VM's are not working, agent state is offline. I'll try to reproduce this whole setup but use Ubuntu 22.04 instead, just to see if I get the same issues.

[weizhouapache]

@sjanssen15
it would be good to share /var/log/cloud.log in system vms
the issue might be caused by a global setting, or some network configurations

re-setup might not help

[sjanssen15]

Just finished the reinstall with Ubuntu 22.04, exact same errors. I will try to get a system VM running and get the log locally from the kvm instance. I'll post it here soon

[sjanssen15]

I had a look at the logging and I saw some errors regarding "Certificate for <127.0.0.1> doesn't match any of the subject alternative names". I remember that the FQDN record is still responding the short name. I changed /etc/hosts and commented the localhost record. Now I get the full FQDN back. Rebooted the server and repeated the setup. This time I still get that DHCP error but now the setup completes and I have 2 running system vm's with agent active.

I connected to the system VM console and I'm able to reach the storage network and management network. Are you still interested in certain logging?

[DaanHoogland]

Hey @sjanssen15 if you think the logs report a false positive yes, otherwise I think we should consider this closed with your prior comment.