:bug: Panic in controller during guest customization

**What steps did you take and what happened?**
This was introduced after the PR https://github.com/vmware-tanzu/vm-operator/pull/167 was merged. 

**What did you expect to happen?**
No controller panic when running guest os customization.

**Is there anything else you would like to add?**
Error stack trace observed as below:
```
I0712 10:39:08.886378       1 controller.go:118]  "msg"="Observed a panic in reconciler: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason" "VirtualMachine"={"name":"vm-w89h","namespace":"vmsvc-e2e-wkwzz9"} "controller"="virtualmachine" "controllerGroup"="vmoperator.vmware.com" "controllerKind"="VirtualMachine" "name"="vm-w89h" "namespace"="vmsvc-e2e-wkwzz9" "reconcileID"="276a9abc-852f-40a5-86b0-333ee297d49d"
panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason [recovered]
panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason [recovered]
        panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason
panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason [recovered]
        panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason
        panic: interface conversion: error is *session.ErrWithReason, not session.ErrWithReason

goroutine 2160 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fa
panic({0x1ea5700, 0xc00085db00})
        /tmp/runner/custom-build/linux64/src/runtime/panic.go:884 +0x213
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.customizeCloudInit({{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x0?, 0x0?, {0xc0011a4580, 0x0, ...})
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_customization.go:304 +0x254

github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).customize(0x229bc9f?, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x229bc9f?, 0xc0009a4c00, {0xc0011a4580, ...})
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_customization.go:332 +0x44e
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).prepareVMForPowerOn(0xc0016e9310, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x40dc2a?, 0xc0006cf100?, 0xc0010f8820)
goroutine 2160 [running]:
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_update.go:685 +0x296
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).UpdateVirtualMachine(0xc0016e9310, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0xc00037ad20, 0xc0016e9348)
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_update.go:918 +0xa25
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fa
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere.(*vSphereVMProvider).updateVirtualMachine(0xc000a3c640, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0xc00002d3b0?, 0xc00002d3b0)
panic({0x1ea5700, 0xc00085db00})
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/vmprovider_vm.go:317 +0x1f4
        /tmp/runner/custom-build/linux64/src/runtime/panic.go:884 +0x213
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere.(*vSphereVMProvider).CreateOrUpdateVirtualMachine(0xc000ede7f8?, {0x25e03d0, 0xc001881530}, 0xc000ede600)
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.customizeCloudInit({{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x0?, 0x0?, {0xc0011a4580, 0x0, ...})
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/vmprovider_vm.go:86 +0x36a
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_customization.go:304 +0x254
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine.(*Reconciler).ReconcileNormal(0xc0001f0d20, 0xc001881530)
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).customize(0x229bc9f?, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x229bc9f?, 0xc0009a4c00, {0xc0011a4580, ...})
        /tmp/runner/cayman_vm-operator/vm-operator/src/controllers/virtualmachine/virtualmachine_controller.go:478 +0x232
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_customization.go:332 +0x44e
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine.(*Reconciler).Reconcile(0xc0001f0d20, {0x25e00f8, 0xc001881470}, {{{0xc0019ea600?, 0x0?}, {0xc0019ea5f0?, 0x40e007?}}})
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).prepareVMForPowerOn(0xc0016e9310, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0x40dc2a?, 0xc0006cf100?, 0xc0010f8820)
        /tmp/runner/cayman_vm-operator/vm-operator/src/controllers/virtualmachine/virtualmachine_controller.go:390 +0x566
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_update.go:685 +0x296
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x25e00f8?, {0x25e00f8?, 0xc001881470?}, {{{0xc0019ea600?, 0x1d79380?}, {0xc0019ea5f0?, 0x0?}}})
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere/session.(*Session).UpdateVirtualMachine(0xc0016e9310, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0xc00037ad20, 0xc0016e9348)
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xc8
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/session/session_vm_update.go:918 +0xa25
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001cf9a0, {0x25e0050, 0xc000a3c4b0}, {0x1f947c0?, 0xc0014a43a0?})
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere.(*vSphereVMProvider).updateVirtualMachine(0xc000a3c640, {{0x25e00f8, 0xc001881d70}, {{0x25e3628, 0xc00151a5f0}, 0x0}, 0xc000ede600}, 0xc00002d3b0?, 0xc00002d3b0)
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x35f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001cf9a0, {0x25e0050, 0xc000a3c4b0})
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x85
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/vmprovider_vm.go:317 +0x1f4
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
github.com/vmware-tanzu/vm-operator/pkg/vmprovider/providers/vsphere.(*vSphereVMProvider).CreateOrUpdateVirtualMachine(0xc000ede7f8?, {0x25e03d0, 0xc001881530}, 0xc000ede600)
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x587
        /tmp/runner/cayman_vm-operator/vm-operator/src/pkg/vmprovider/providers/vsphere/vmprovider_vm.go:86 +0x36a
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine.(*Reconciler).ReconcileNormal(0xc0001f0d20, 0xc001881530)
        /tmp/runner/cayman_vm-operator/vm-operator/src/controllers/virtualmachine/virtualmachine_controller.go:478 +0x232
github.com/vmware-tanzu/vm-operator/controllers/virtualmachine.(*Reconciler).Reconcile(0xc0001f0d20, {0x25e00f8, 0xc001881470}, {{{0xc0019ea600?, 0x0?}, {0xc0019ea5f0?, 0x40e007?}}})
        /tmp/runner/cayman_vm-operator/vm-operator/src/controllers/virtualmachine/virtualmachine_controller.go:390 +0x566
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x25e00f8?, {0x25e00f8?, 0xc001881470?}, {{{0xc0019ea600?, 0x1d79380?}, {0xc0019ea5f0?, 0x0?}}})
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0001cf9a0, {0x25e0050, 0xc000a3c4b0}, {0x1f947c0?, 0xc0014a43a0?})
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x35f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0001cf9a0, {0x25e0050, 0xc000a3c4b0})
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
        /tmp/runner/home/mts/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x587
```


**Please tell us about your environment.**

<table>
<tr>
<td></td>
<th>Value</th>
<th>How to Obtain</th>
</tr>
<tr>
<th>Supervisor version</th>
<td><code></code></td>
<td><code>rpm -qa VMware-wcp</code> on the vCenter appliance</td>
</tr>
<tr>
<th>Supervisor node image version</th>
<td><code></code></td>
<td><code>rpm -qa VMware-wcpovf</code> on the vCenter appliance</td>
</tr>
<tr>
<th>Kubernetes version</th>
<td><code></code></td>
<td><code>kubectl version</code></td>
</tr>
<tr>
<th>VM Operator version</th>
<td><code></code></td>
<td>

```shell
kubectl -n vmware-system-vmop get pods \
  -ojsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}' | \
  sort -u | \
  grep vmop | \
  awk -F'/' '{print $3}' | \
  awk -F: '{print $2}'
```

</td>
</tr>
</table>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

🐛 Panic in controller during guest customization #177

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	Value	How to Obtain
Supervisor version		`rpm -qa VMware-wcp` on the vCenter appliance
Supervisor node image version		`rpm -qa VMware-wcpovf` on the vCenter appliance
Kubernetes version		`kubectl version`
VM Operator version		kubectl -n vmware-system-vmop get pods \ -ojsonpath='{range .items[].spec.containers[]}{.image}{"\n"}{end}' \| \ sort -u \| \ grep vmop \| \ awk -F'/' '{print $3}' \| \ awk -F: '{print $2}'

🐛 Panic in controller during guest customization #177

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions