vsphere_host_virtual_switch resource attempts to change the VMNIC order

[erSitzt]

Community Guidelines

• I have read and agree to the HashiCorp Community Guidelines .
• Vote on this issue by adding a 👍 reaction to the original issue initial description to help the maintainers prioritize.
• Do not leave "+1" or other comments that do not add relevant information or questions.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Terraform

v1.5.6

Terraform Provider

v2.4.3

VMware vSphere

v7.0.3

Description

Same problem as in #1503

Current vmnic order is

But terraform wants to update the order on multiple servers.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.vswitch["esxi-4.json"].vsphere_host_virtual_switch.iscsi-switch will be updated in-place
  ~ resource "vsphere_host_virtual_switch" "iscsi-switch" {
        id                        = "tf-HostVirtualSwitch:host-643727:vSwitch-iSCSI"
        name                      = "vSwitch-iSCSI"
      ~ network_adapters          = [
          - "vmnic2",
            "vmnic0",
          + "vmnic2",
        ]
        # (19 unchanged attributes hidden)
    }

  # module.vswitch["esxi-5.json"].vsphere_host_virtual_switch.iscsi-switch will be updated in-place
  ~ resource "vsphere_host_virtual_switch" "iscsi-switch" {
        id                        = "tf-HostVirtualSwitch:host-637250:vSwitch-iSCSI"
        name                      = "vSwitch-iSCSI"
      ~ network_adapters          = [
          - "vmnic2",
            "vmnic0",
          + "vmnic2",
        ]
        # (19 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Fun fact:
I have one server where the vmnic order is the other way round

...and terraform is not trying to reorder the vmnics on this host

To be clear, the current vmnic order on all of those hosts currently matches what is configured in terraform.

Affected Resources or Data Sources

resource/vsphere_host_virtual_switch

Terraform Configuration

Nothing special here i think...

https://gist.github.com/erSitzt/6f7176babcefe6dab8464a3e48e9cda5

The json used here looks like this

{
    "esx-nummer"                :   4,
    "dnsname"                   :   "esxi-4.domain.com",
    "configname"                :   "esxi-4.json",
    "iscsi_1-ip"                :   "192.168.101.204",
    "iscsi_1-nic"               :   "vmnic0",
    "iscsi_2-ip"                :   "192.168.102.204",
    "iscsi_2-nic"               :   "vmnic2"
}

Debug Output

https://gist.github.com/erSitzt/bd4661e41f42c054516ff8decfe7449d

Panic Output

No response

Expected Behavior

No change detected..

Actual Behavior

Change detected on every plan/apply run

Steps to Reproduce

Not sure, will add info if i can test more

Environment Details

No response

Screenshots

No response

References

No response

[github-actions]

Hello, erSitzt! 🖐

Thank you for submitting an issue for this provider. The issue will now enter into the issue lifecycle.

If you want to contribute to this project, please review the contributing guidelines and information on submitting pull requests.

[erSitzt]

quick note, i switched cables on the host , where the order was different`and adjusted the json in my terraform project, now terraform detects a change on this host on every run as well.

So it might have something to do with vmnic0 being first in the list ?

[spacegospod]

Hey @erSitzt the only way I was able to reproduce this was to change the failover order outside TF and attempt to plan/apply the original configuration again.

There is no special handling in the provider in terms of adapter ordering and there's definitely nothing special about vmnic0.
Can you confirm that nobody and nothing apart from Terraform is making changes to this host?

[erSitzt]

i will recheck this tomorrow, but im pretty sure this was a change that terraform would want to do on every run, even if they were only seconds apart with no other process/user changing any settings in vmware.

...but let be verify if this is still happening with current versions...

[erSitzt]

@spacegospod Still happening with current version v2.8.2

It still happens for all hosts after some time.. not sure what is triggering it.
BUT i can provoke this bei entering and exiting maintenance mode for a host.
Tried this for 3 out of 8 hosts and all 3 showed up in terraform plan after cycling maintenance mode.

[spacegospod]

@spacegospod Still happening with current version v2.8.2

It still happens for all hosts after some time.. not sure what is triggering it. BUT i can provoke this bei entering and exiting maintenance mode for a host. Tried this for 3 out of 8 hosts and all 3 showed up in terraform plan after cycling maintenance mode.

Thanks for re-testing. Is the order of the active adapters messed up in the vCenter UI too?

[erSitzt]

Nope vCenter is always in the same order.
I noticed that after some time all hosts show up with terraform plan, but not all at the same time.
We have a very basic VMware setup, no 3rd party integrations used to configure stuff and no other admins changing stuff while this is happening.

[spacegospod]

Looks like entering maintenance mode was the missing piece of the puzzle. I was able to reproduce the bug in 2 out of 6 attempts, it doesn't happen every time on my setup.

I noticed that after exiting maintenance mode the adapters are flipped on the HostVirtualSwitchBondBridge but are left intact on the HostNetworkPolicy.
I don't know why the terraform provider uses the bridge property but it is definitely wrong. It should rely on the network policy whenever ordering is of importance.

References:
https://developer.broadcom.com/xapis/virtual-infrastructure-json-api/8.0.3/data-structures/HostVirtualSwitchBondBridge
https://developer.broadcom.com/xapis/virtual-infrastructure-json-api/8.0.3/data-structures/HostNicOrderPolicy

[erSitzt]

@spacegospod nice :) thanks for investigating

Funny thing why this is happening for all hosts after some time.. first i suspected it had something to do with ha primary election, because that was the first host i sent into maintenance.

[spacegospod]

Hey @erSitzt I got back to this issue and it turned out I wasn't quite right about the bond bridge.

You see, the active and standby nics are stored in order on the policy object but the only structure that contains the complete set of adapters is still the bond bridge. That means that the only way to identify the assigned but unused adapters is to diff the network_adapters property with active_nics and standby_nics.

It looks like the vCenter UI sorts the unused adapters list. For example if you add "vmnic2" and "vmnic1" (in that order) to a swtich and then open up the edit adapters dialog again it would list them as "vmnic1", "vmnic2".
Now with Terraform we don't have that luxury since we keep the actual ordered array in the state file and we detect the diff.

In practice the order of the unused adapters doesn't matter since they don't take part in any network traffic.
What we could do in the TF provider is to diff the array we read from the bond bridge and ignore any sorting changes as long as the set we have in state and the one we get from the server match.

A simpler alternative would be to just ignore any changes to network_adapters in the TF config like so

  lifecycle {
    ignore_changes = [
      "network_adapters",
    ]
  }

Is that a viable option for you? I'm not sure if the ordering change after exiting maintenance mode is a bug or not and I don't know if it's reproducible in newer versions of vSphere. I tried with 8.0.3 today and I haven't been able to observe this yet.

[spacegospod]

I filed a PR that adds a custom diff check for network_adapters which ignores the order of elements, just in case we decide to go down that road #2388

[erSitzt]

thanks @spacegospod @tenthirtyam

[github-actions]

This functionality has been released in v2.13.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!