4.20.1 Can't attach addon disk to VM running on VMware - vCenter #11094

leduyquy · 2025-06-23T02:56:49Z

leduyquy
Jun 23, 2025

problem

In new version Cloudstack: 4.20.1.0, I see it support VMware 8U3 so I did upgrade vCenter to 8.0.3.00500, But affter upgrade vCenter i testing attach disk to VM running on it. everything is normal when i rollback vcenter to old version:

Log on management:

2025-06-23 09:50:37,646 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:167cef34) Add job-132862 into job monitoring
2025-06-23 09:50:37,649 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp2038105753-51180:[ctx-356feba0, ctx-fb5487f9]) (logid:0d6e0b53) submit async job-132862, details: AsyncJob {"accountId":10,"cmd":"org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd","cmdInfo":"{"virtualmachineid":"5007e13f-28e0-4f9e-b191-428c381c9288","response":"json","ctxUserId":"60","sessionkey":"aZTCm6g7HBWrL9uH08ocEqYxnl8","httpmethod":"GET","ctxStartEventId":"1265802","id":"d4a8082f-d4e4-415f-878e-667cba85d103","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"d4a8082f-d4e4-415f-878e-667cba85d103\",\"interface com.cloud.vm.VirtualMachine\":\"5007e13f-28e0-4f9e-b191-428c381c9288\"}","ctxAccountId":"10","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","cmdEventType":"VOLUME.ATTACH"}","cmdVersion":0,"completeMsid":null,"created":null,"id":132862,"initMsid":345050445235,"instanceId":4274,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":60,"uuid":"0654201d-f27e-4b7d-8d64-0a067bc74a80"}
2025-06-23 09:50:37,650 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Executing AsyncJob {"accountId":10,"cmd":"org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd","cmdInfo":"{"virtualmachineid":"5007e13f-28e0-4f9e-b191-428c381c9288","response":"json","ctxUserId":"60","sessionkey":"aZTCm6g7HBWrL9uH08ocEqYxnl8","httpmethod":"GET","ctxStartEventId":"1265802","id":"d4a8082f-d4e4-415f-878e-667cba85d103","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"d4a8082f-d4e4-415f-878e-667cba85d103\",\"interface com.cloud.vm.VirtualMachine\":\"5007e13f-28e0-4f9e-b191-428c381c9288\"}","ctxAccountId":"10","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","cmdEventType":"VOLUME.ATTACH"}","cmdVersion":0,"completeMsid":null,"created":null,"id":132862,"initMsid":345050445235,"instanceId":4274,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":60,"uuid":"0654201d-f27e-4b7d-8d64-0a067bc74a80"}
2025-06-23 09:50:37,658 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Access to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] granted to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] by DomainChecker on behalf of user <>
2025-06-23 09:50:37,662 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Access to VM {"id":1544,"instanceName":"i-10-1544-VM","type":"User","uuid":"5007e13f-28e0-4f9e-b191-428c381c9288"} granted to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] by DomainChecker on behalf of user <>
2025-06-23 09:50:37,664 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Account [Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}]] has access to resource.
2025-06-23 09:50:37,690 DEBUG [c.c.h.d.HypervisorCapabilitiesDaoImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Hypervisor capabilities for hypervisor: VMware, version: 8.0.2.0 can not be found. Trying to find capabilities for the parent version: 8.0.2
2025-06-23 09:50:37,696 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Access to VolumeObject {"dataStore":"StoragePool {"id":2,"name":"vsanDatastore","poolType":"PreSetup","uuid":"7baf972d-35f4-394c-b9c3-0c3e13b4144a"}","volumeVO":"Volume {"id":4274,"instanceId":null,"name":"data","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","volumeType":"DATADISK"}"} granted to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] by DomainChecker on behalf of user <>
2025-06-23 09:50:37,697 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Access to VM instance {"id":1544,"instanceName":"i-10-1544-VM","state":"Running","type":"User","uuid":"5007e13f-28e0-4f9e-b191-428c381c9288"} granted to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] by DomainChecker on behalf of user <>
2025-06-23 09:50:37,703 INFO [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Trying to attach volume [VolumeObject {"dataStore":"StoragePool {"id":2,"name":"vsanDatastore","poolType":"PreSetup","uuid":"7baf972d-35f4-394c-b9c3-0c3e13b4144a"}","volumeVO":"Volume {"id":4274,"instanceId":null,"name":"data","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","volumeType":"DATADISK"}"}] to VM instance [VM instance {"id":1544,"instanceName":"i-10-1544-VM","state":"Running","type":"User","uuid":"5007e13f-28e0-4f9e-b191-428c381c9288"}], update async job-132862 [AsyncJob {"accountId":10,"cmd":"org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd","cmdInfo":"{"virtualmachineid":"5007e13f-28e0-4f9e-b191-428c381c9288","response":"json","ctxUserId":"60","sessionkey":"aZTCm6g7HBWrL9uH08ocEqYxnl8","httpmethod":"GET","ctxStartEventId":"1265802","id":"d4a8082f-d4e4-415f-878e-667cba85d103","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"d4a8082f-d4e4-415f-878e-667cba85d103\",\"interface com.cloud.vm.VirtualMachine\":\"5007e13f-28e0-4f9e-b191-428c381c9288\"}","ctxAccountId":"10","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","cmdEventType":"VOLUME.ATTACH"}","cmdVersion":0,"completeMsid":null,"created":null,"id":132862,"initMsid":345050445235,"instanceId":4274,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":60,"uuid":"0654201d-f27e-4b7d-8d64-0a067bc74a80"}] progress status
2025-06-23 09:50:37,705 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Update async-job attachment, job-132862, instanceType: Volume, instanceId: 4274
2025-06-23 09:50:37,707 DEBUG [c.c.a.ApiServer] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Retrieved cmdEventType from job info: VOLUME.ATTACH
2025-06-23 09:50:37,733 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Sync job-132863 execution on object VmWorkJobQueue.1544
2025-06-23 09:50:37,762 DEBUG [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) New job VmWorkJobVO : {, step: Starting, vmType: Instance, vmInstanceId: 1544, secondaryObjectIdentifier: nullAsyncJob {"accountId":10,"cmd":"com.cloud.vm.VmWorkAttachVolume","cmdInfo":"rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrQXR0YWNoVm9sdW1lB62v-WGH4hwCAAJMAAhkZXZpY2VJZHQAEExqYXZhL2xhbmcvTG9uZztMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAoAAAAAAAAAPAAAAAAAAAYIdAAUVm9sdW1lQXBpU2VydmljZUltcGxwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAABCy","cmdVersion":0,"completeMsid":null,"created":null,"id":132863,"initMsid":345050445235,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":60,"uuid":"8d75d03d-44d6-463a-9150-94f7f6589203"}}, result field: null
2025-06-23 09:50:40,786 WARN [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Skipping delivering event Event {"description":"{"details":"Volume Id: d4a8082f-d4e4-415f-878e-667cba85d103 VmId: 5007e13f-28e0-4f9e-b191-428c381c9288","event":"VOLUME.ATTACH","status":"Completed"}","eventId":null,"eventType":"VOLUME.ATTACH","eventUuid":null,"resourceType":"Volume","resourceUUID":null} to any webhook as account ID is missing
2025-06-23 09:50:40,787 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Failed to publish event [category: ActionEvent, type: VOLUME.ATTACH] on bus webhookEventBus
2025-06-23 09:50:40,802 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Unexpected exception while executing org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd com.cloud.utils.exception.CloudRuntimeException: Failed to attach volume data to VM <>
2025-06-23 09:50:40,803 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Complete async job-132862, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to attach volume data to VM <>"}
2025-06-23 09:50:40,803 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Publish async job-132862 complete on message bus
2025-06-23 09:50:40,804 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Wake up jobs related to job-132862
2025-06-23 09:50:40,804 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Update db status for job-132862
2025-06-23 09:50:40,805 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Wake up jobs joined with job-132862 and disjoin all subjobs created from job- 132862
2025-06-23 09:50:40,813 DEBUG [c.c.a.ApiServer] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Retrieved cmdEventType from job info: VOLUME.ATTACH
2025-06-23 09:50:40,817 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Done executing org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for job-132862
2025-06-23 09:50:40,817 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Remove job-132862 from job monitoring

Since i had to rollback to vcenter to secure workload i can only provide more information from old log

versions

Cloudstack: 4.20.1.0
vCenter: 8.0.3.00500, Build number: 24674346
VMware ESXi : 8.0.2, 22380479

The steps to reproduce the bug

Upgrade vCenter from 8.0.0 to 8.0.3
Attach addon disk VM
Error
...

What to do about it?

Attach addon disk working normal

sureshanaparti · 2025-06-23T09:30:17Z

sureshanaparti
Jun 23, 2025
Collaborator

@leduyquy any errors noticed in vcenter tasks / events during attach volume operation?

0 replies

kiranchavala · 2025-06-23T10:18:09Z

kiranchavala
Jun 23, 2025
Collaborator

@leduyquy, what type of primary storage are you using

I have tested with Nfs and the volume attachment works fine on 4.20.1 and 80u3

0 replies

leduyquy · 2025-06-23T10:26:02Z

leduyquy
Jun 23, 2025
Author

I using vsan for primary storage, I don't see any error during attach volume operator (I'm not sure about it either because I've rolled back it to old version)

0 replies

sureshanaparti · 2025-06-23T12:27:52Z

sureshanaparti
Jun 23, 2025
Collaborator

ok @leduyquy , I've verified attach volume with NFS primary storage, in 4.20.1 environment and VMware vCenter: 8.0.3.00500, Build number: 24674346 (vCenter Server 8.0 Update 3e); VMware ESXi : 8.0.3, 24674464 (ESXi 8.0 P05 / ESXi 8.0 Update 3e). No issues noticed.

0 replies

winterhazel · 2025-06-23T16:39:16Z

winterhazel
Jun 23, 2025
Collaborator

@leduyquy could you check:

in CloudStack, what values do the settings rootDiskController and dataDiskController have in your VM?
in vCenter, what disk controllers does your VM have?

0 replies

leduyquy · 2025-06-24T09:28:26Z

leduyquy
Jun 24, 2025
Author

Hi @winterhazel

in CloudStack, what values do the settings rootDiskController and dataDiskController have in your VM?

I set rootDiskController and dataDiskController to pvscsi

in vCenter, what disk controllers does your VM have?

at vcenter it is set: VMware Paravirtual

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

4.20.1 Can't attach addon disk to VM running on VMware - vCenter #11094

Uh oh!

{{title}}

Uh oh!

Replies: 6 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

4.20.1 Can't attach addon disk to VM running on VMware - vCenter #11094

Uh oh!

leduyquy Jun 23, 2025

problem

versions

The steps to reproduce the bug

What to do about it?

Replies: 6 comments

Uh oh!

sureshanaparti Jun 23, 2025 Collaborator

Uh oh!

kiranchavala Jun 23, 2025 Collaborator

Uh oh!

leduyquy Jun 23, 2025 Author

Uh oh!

sureshanaparti Jun 23, 2025 Collaborator

Uh oh!

winterhazel Jun 23, 2025 Collaborator

Uh oh!

Uh oh!

leduyquy Jun 24, 2025 Author

leduyquy
Jun 23, 2025

sureshanaparti
Jun 23, 2025
Collaborator

kiranchavala
Jun 23, 2025
Collaborator

leduyquy
Jun 23, 2025
Author

sureshanaparti
Jun 23, 2025
Collaborator

winterhazel
Jun 23, 2025
Collaborator

leduyquy
Jun 24, 2025
Author