Error while adding Local ISO.

[gjosh91]

ERROR:
Request Failed (530)
There is no secondary storage VM for downloading template to image store secondary.

My Setup:
All running on Ubuntu 22
Management Server : IP - 192.168.1.122
NFS Server : IP - 192.168.1.182
KVM - 192.168.1.44

Required :
Add ISO from local storage .

Just for your perspective . The dashboard is showing my System VMs are running one is console proxy and the other is secondary storage vm.
My infrastructure is showing that I have a primary storage and a secondary storage. My Dashboard is showing the memory allocated to my primary storage but not to my secondary storage.
My mount point are also functioning and no error in log.

Any guidance will be appreciated . I have also attached the screenshot for agent logs and management logs for your reference.

Management server log -

Please note - I do not think the error related to certificate is cause for the error I am receiving. Correct me if I am wrong.
2024-07-25 12:46:16,560 DEBUG [c.c.u.s.Script] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Executing command [/bin/bash -c du -sh /var/log/cloudstack/management/management-server.log | cut -f '1' ].
2024-07-25 12:46:16,564 DEBUG [c.c.u.s.Script] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Successfully executed process [17716] for command [/bin/bash -c du -sh /var/log/cloudstack/management/management-server.log | cut -f '1' ].
2024-07-25 12:46:16,565 DEBUG [c.c.u.s.Script] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Executing command [/bin/bash -c df -h /var/log/cloudstack/management/management-server.log | grep -v Filesystem | awk '{print "on disk " $1 " mounted on " $6 " (" $5 " full)"}' ].
2024-07-25 12:46:16,571 DEBUG [c.c.u.s.Script] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Successfully executed process [17719] for command [/bin/bash -c df -h /var/log/cloudstack/management/management-server.log | grep -v Filesystem | awk '{print "on disk " $1 " mounted on " $6 " (" $5 " full)"}' ].
2024-07-25 12:46:16,573 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersdirect.capacity, 122261
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersdirect.count, 20
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersdirect.used, 122261
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersmapped.capacity, 76568
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersmapped.count, 5
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail buffersmapped.used, 76568
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail gcPS-MarkSweep.count, 6
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail gcPS-MarkSweep.time, 1147
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail gcPS-Scavenge.count, 33
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail gcPS-Scavenge.time, 249
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail jvmname, 15681@hcimg-HP-ProBook-440-G5
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail jvmuptime, 1409565
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail jvmvendor, Ubuntu OpenJDK 64-Bit Server VM 11.0.23+9-post-Ubuntu-1ubuntu122.04.1 (11)
2024-07-25 12:46:16,574 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memoryheap.committed, 953155584
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memoryheap.init, 130023424
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memoryheap.usage, 0.1845245948043442
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.committed, 194682880
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.init, 7667712
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.max, -1
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.usage, -1.89691784E8
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorynon-heap.used, 189691784
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.CodeHeap-'non-nmethods'.usage, 0.30991228070175436
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.CodeHeap-'non-profiled-nmethods'.usage, 0.14020948580378564
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.CodeHeap-'profiled-nmethods'.usage, 0.2614160862465425
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.Compressed-Class-Space.usage, 0.011386938393115997
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.Metaspace.usage, 0.9769060697991255
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.PS-Eden-Space.usage, 0.31128890661153935
2024-07-25 12:46:16,575 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.PS-Old-Gen.usage, 0.1051116015875309
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorypools.PS-Survivor-Space.usage, 0.1280120481927711
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorytotal.committed, 1147838464
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorytotal.init, 137691136
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorytotal.max, 1908932607
2024-07-25 12:46:16,576 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail memorytotal.used, 541936800
2024-07-25 12:46:16,577 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail threadsdeadlocks, []
2024-07-25 12:46:16,577 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail threadsnew.count, 0
2024-07-25 12:46:16,578 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) not storing detail threadstimed_waiting.count, 74
2024-07-25 12:46:16,581 DEBUG [c.c.c.ClusterManagerImpl] (StatsCollector-3:ctx-70214bde) (logid:2cea8634) Forwarding {"managementServerHostId":1,"managementServerHostUuid":"fc5923e9-3eac-46b0-93b3-3edbf9b595fb","collectionTime":"Jul 25, 2024, 12:46:16 PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":953155584,"freeJvmMemoryBytes":602002456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1409455,"jvmStartTime":1721890367082,"availableProcessors":8,"loadAverage":0.435546875,"totalInit":137691136,"totalUsed":540842440,"totalCommitted":1147838464,"pid":15681,"jvmName":"15681@hcimg-HP-ProBook-440-G5","jvmVendor":"Ubuntu","jvmVersion":"11.0.23+9-post-Ubuntu-1ubuntu122.04.1","osDistribution":"Ubuntu 22.04.4 LTS","agentCount":1,"heapMemoryUsed":352245016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":11,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":112,"systemMemoryTotal":8206192640,"systemMemoryFree":700325888,"systemMemoryUsed":1294708,"systemMemoryVirtualSize":8336060416,"logInfo":"/var/log/cloudstack/management/management-server.log using: 34M\non disk /dev/nvme0n1p2 mounted on / (11% full)","systemTotalCpuCycles":13586.599,"systemLoadAverages":[0.44,0.49,0.58],"systemCyclesUsage":[352458,122206,5591261],"dbLocal":true,"usageLocal":false,"systemBootTime":"Jul 25, 2024, 10:38:22 AM","kernelVersion":"5.15.0-116-generic"} to 233349941812470
2024-07-25 12:46:16,581 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:ctx-9bdd489d) (logid:4e9880d5) Cluster PDU 233349941812470 -> 233349941812470. agent: 0, pdu seq: 24, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"fc5923e9-3eac-46b0-93b3-3edbf9b595fb","collectionTime":"Jul 25, 2024, 12:46:16 PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":953155584,"freeJvmMemoryBytes":602002456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1409455,"jvmStartTime":1721890367082,"availableProcessors":8,"loadAverage":0.435546875,"totalInit":137691136,"totalUsed":540842440,"totalCommitted":1147838464,"pid":15681,"jvmName":"15681@hcimg-HP-ProBook-440-G5","jvmVendor":"Ubuntu","jvmVersion":"11.0.23+9-post-Ubuntu-1ubuntu122.04.1","osDistribution":"Ubuntu 22.04.4 LTS","agentCount":1,"heapMemoryUsed":352245016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":11,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":112,"systemMemoryTotal":8206192640,"systemMemoryFree":700325888,"systemMemoryUsed":1294708,"systemMemoryVirtualSize":8336060416,"logInfo":"/var/log/cloudstack/management/management-server.log using: 34M\non disk /dev/nvme0n1p2 mounted on / (11% full)","systemTotalCpuCycles":13586.599,"systemLoadAverages":[0.44,0.49,0.58],"systemCyclesUsage":[352458,122206,5591261],"dbLocal":true,"usageLocal":false,"systemBootTime":"Jul 25, 2024, 10:38:22 AM","kernelVersion":"5.15.0-116-generic"}
2024-07-25 12:46:16,581 DEBUG [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:ctx-9bdd489d) (logid:4e9880d5) Executing ClusterServicePdu with service URL: https://127.0.0.1:9090/clusterservice
2024-07-25 12:46:16,585 ERROR [c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:ctx-9bdd489d) (logid:4e9880d5) Exception from : https://127.0.0.1:9090/clusterservice, method : null, exception :
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <127.0.0.1> doesn't match any of the subject alternative names: [fe80:0:0:0:da73:d3bc:cbd3:976e, 192.168.1.122, hcimg-HP-ProBook-440-G5, cloudstack.internal]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at com.cloud.cluster.ClusterServiceServletImpl.executePostMethod(ClusterServiceServletImpl.java:142)
at com.cloud.cluster.ClusterServiceServletImpl.execute(ClusterServiceServletImpl.java:105)
at com.cloud.cluster.ClusterManagerImpl.onSendingClusterPdu(ClusterManagerImpl.java:276)
at com.cloud.cluster.ClusterManagerImpl$1.runInContext(ClusterManagerImpl.java:236)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-07-25 12:46:16,588 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Worker-5:ctx-9bdd489d) (logid:4e9880d5) Cluster PDU 233349941812470 -> 233349941812470 completed. time: 6ms. agent: 0, pdu seq: 24, pdu ack seq: 0, json: {"managementServerHostId":1,"managementServerHostUuid":"fc5923e9-3eac-46b0-93b3-3edbf9b595fb","collectionTime":"Jul 25, 2024, 12:46:16 PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":953155584,"freeJvmMemoryBytes":602002456,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":1409455,"jvmStartTime":1721890367082,"availableProcessors":8,"loadAverage":0.435546875,"totalInit":137691136,"totalUsed":540842440,"totalCommitted":1147838464,"pid":15681,"jvmName":"15681@hcimg-HP-ProBook-440-G5","jvmVendor":"Ubuntu","jvmVersion":"11.0.23+9-post-Ubuntu-1ubuntu122.04.1","osDistribution":"Ubuntu 22.04.4 LTS","agentCount":1,"heapMemoryUsed":352245016,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":11,"threadsRunnableCount":10,"threadsTerminatedCount":0,"threadsTotalCount":196,"threadsWaitingCount":112,"systemMemoryTotal":8206192640,"systemMemoryFree":700325888,"systemMemoryUsed":1294708,"systemMemoryVirtualSize":8336060416,"logInfo":"/var/log/cloudstack/management/management-server.log using: 34M\non disk /dev/nvme0n1p2 mounted on / (11% full)","systemTotalCpuCycles":13586.599,"systemLoadAverages":[0.44,0.49,0.58],"systemCyclesUsage":[352458,122206,5591261],"dbLocal":true,"usageLocal":false,"systemBootTime":"Jul 25, 2024, 10:38:22 AM","kernelVersion":"5.15.0-116-generic"}

Agent log -

[DaanHoogland]

@gjosh91 , I don't think the cluster certificate error is relevant either but just to be sure, are you running a dual management server setup? If so please stop one of the management servers and try again.

In your dashboard, do you see not only both systemvms running but also both agent-states as connected?

Nothing I see here seems to be relevant to a upload from local storage failing.

Can you check if your secondary storage is properly seeded, i.e. there is a templates directory, and if you upload from url that a template gets created?

[gjosh91]

Hi @DaanHoogland ,

sorry for the late reply.

My agent service is running but in dashboard is not showing as connected.
I am attaching the screenshots of my dashboard for your reference :

1. System VM snapshot

2. My KVM host :

3. My secondary storage and primary storage :

Please guide how to make my agent state connected ?

Also regarding the template directory :

My template directory is getting created on the nfs machine and I can see the system vm template there in the directory structure /mnt/secondary/template/tmpl/1/3
but not the template I created using the url source. Maybe because I am using direct download which is skipping the secondary storage. As when I am not using the direct download option I am getting the previous error of 503 no secondary storage to download systemvm template
Here is the screenshot of my template from dashboard :

Also please notice that although I can see the secondary storage , in the dashboard the secondary storage allocated space is being reflected as 0 Gb :

When trying to create instance using the default system VM template . I get this :

Just to add to the above query :

My VM gets created successfully when doing direct download but my instance does not get created . Getting this error :

Error in logs :

2024-07-29 17:11:39,393 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-3:ctx-deee2efb job-273/job-274 ctx-1d15c856)
(logid:6628bc3b) Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException:
Unable to create a deployment for VM instance
{"id":13,"instanceName":"i-2-13-VM","type":"User","uuid":"639eecd4-dd5d-4168-b229-708f504d0a68"}
Scope=interface com.cloud.dc.DataCenter; id=1

---

Error
Unable to start a VM [639eecd4-dd5d-4168-b229-708f504d0a68] due to [Unable to create a deployment for VM instance {"id":13,"instanceName":"i-2-13-VM","type":"User","uuid":"639eecd4-dd5d-4168-b229-708f504d0a68"}].

[gjosh91]

Hi @DaanHoogland Sorry for pestering you. Hope you are doing fine.
Can you please guide further . Is there something that I am doing wrong?

[DaanHoogland]

Het @gjosh91 , sorry for not answering, in visiting the Olympics. I'll have look later. In the meanwhile, check if you can mount the storages on your hosts and ssvm.

[gjosh91]

@DaanHoogland Hope you are doing great. I was away from this topic for a while. I have gone through all the possibles steps I could have .
As per my understanding which might be limited as I am new to cloud I am not able to add the ISO/Template using secondary storage.
Also after reading through your previous comment I wanted to update that in the dashboar of Sytem VM I can see both my VMs in running state but the Agent state is not UP.
I would also like to know that is there any documentation /book /article which I can refer to understand how the agent and management are communicating internally .
Thank you for any guidance you can provide.

[DaanHoogland]

@gjosh91 , I would go through the page that Jithin pointed to first. You are right that it might not be completely up to date but most of it is valid. Please suggest any changes if you find errors.
Espaecially the /usr/local/cloud/systemvm/ssvm-check.sh script is useful in pointing to DNS or routing errors.
the console proxy not bein up is probably secondary fallout, so focus on getting the SSVM to function first.

[rajujith]

@gjosh91 can you try the troubleshooting steps here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting

[gjosh91]

@rajujith I tried the troubleshooting steps but there are two issues :
first the document is not up to date with the current version
second I did not find any specific resolution for my problem.
Although I tried some other steps like health checkup and secondary storage allocation is low but to no avail.

[rajujith]

@gjosh91 Yes that document is not upto date but the common troubleshooting steps remains the same mostly.
In your installation you should get the agent in the system VMs connected to the management server on port 8250. You can check the '/var/log/cloud.log' in the System VMs and management-server.log for any error during the connection attempts. You should also check for port 8250 accessibility on management server from the system VMs. Make sure the global configuration parameter 'host' has a value that points to the management server IP/hostname.