VM console cannot access after VNC certificate expired

[havengit]

ISSUE TYPE

• Bug Report

COMPONENT NAME

CPVM

CLOUDSTACK VERSION

4.18 

CONFIGURATION

advanced networking

OS / ENVIRONMENT

Hypervisro: KVM virtualization with ubuntu 22.04

SUMMARY

VM console cannot access after VNC certificate expired。
Default certificate valid for 1 year ，then automatically renews certificates when they expire by cloudstack。
I made sure the certificate was updated on the host，but Th running VM instances need stop and start to apply the new certificate or migraton to other host. For production environments, this is hard to do

I don't know how to get a new certificate to take effect without rebooting the vm

#7015

STEPS TO REPRODUCE

EXPECTED RESULTS

Make a new certificate to take effect without rebooting the vm
Provide a setting to turn off tls for vnc

ACTUAL RESULTS

[DaanHoogland]

@havengit , I think this is more a lack of feature than a bug, which admittedly comes down to the same to you. So I will mark it as an improvement. Dynamically setting the certificate will not be easy. Did you try rebooting just the console proxy? I would guess that is the actually needed action.

[weizhouapache]

there are some global settings ("ca.framework.*"), including

• ca.framework.background.task.delay
• ca.framework.cert.automatic.renewal
• ca.framework.cert.expiry.alert.period

if auto renewal is enabled, the agent cert should be auto-renewed.

otherwise you can do it manually . refer to #9562 (comment)

[havengit]

Yes , I have restarted cpvm and libvirt on host. I confirm that the certificate has been successfully updated on the host. May be same bug in qemu. After stop vm and start vm console works fine .

[weizhouapache]

Yes , I have restarted cpvm and libvirt on host. I confirm that the certificate has been successfully updated on the host. May be same bug in qemu. After stop vm and start vm console works fine .

@havengit
is it necessary to stop/start vm if you have restarted libvirtd and cloudstack-agent ?
if so, it has big impact

[havengit]

@weizhouapache Restart libvirtd and cloudstack-agent has no effect ，VM must be rebooted or live migration to other host .
I don't know if anyone else has encountered this, but this feature should have been added in 4.18, so maybe not many people are using it.

[havengit]

Using virt-viewer also fails to connect, so it shouldn't be an ACS issue.

[weizhouapache]

this may be related to #7015

can you check if

• vnc_tls is set to 1 in /etc/libvirt/qemu.conf ?
• console works after live migration to another host ?

[havengit]

Yes , the vnc tls were enabled in all host. It seems that the qemu process that is running, does not recognize the certificate change and still uses the old certificate. Stop the startup or migrate and the new qemu process will use the new certificate.
vnc_tls=1
vnc_tls_x509_verify=1
vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc"

[weizhouapache]

Yes , the vnc tls were enabled in all host. It seems that the qemu process that is running, does not recognize the certificate change and still uses the old certificate. Stop the startup or migrate and the new qemu process will use the new certificate. vnc_tls=1 vnc_tls_x509_verify=1 vnc_tls_x509_cert_dir="/etc/pki/libvirt-vnc"

thanks @havengit
good to know that migration fixes the issue. stopping/starting all vms is not possible for large production environments.

To summarize,

• vm console becomes unavailable if cloudstack-agent certificates expires and is renewed.
• it is a qemu issue. ACS cannot do anything to fix it.
• users have to migrate or stop/start the VMs so that new qemu process will use the new certificates

cc @DaanHoogland @rohityadavcloud @nvazquez

[havengit]

Thanks ,weizhou and community , I have change Ca framework cert validity period to a very long time . I won't run into this problem in the future.