Database locked errors during iosxe_interface_ethernet parallel creation with source_templates #201
Comments
|
#show netconf-yang datastores #clear netconf-yang session 18 What exactly is going on with this session 18? |
|
Unfortunately, the RESTCONF implementation on XE returns a success response while the database is still locked. Therefore we have to rely on a retry mechanism. I assume it is eventually able to apply the configuration after running "terraform apply" again. If that is the case, the provider configuration has an attribute to change the number of retries, so you can try increasing it accordingly. |
|
This works if I retry 30-40 times. It takes 40 minutes for the stack to converge. This is not a useable workaround. Is there an upstream bug I can follow? If not, there needs to be one. |
|
So you are saying, if you increase the number of retries to 30-40 via the provider configuration, you are able to apply the full configuration in a single run ("terraform apply"), but it takes 40 min to complete? I am not aware of an upstream bug, but feel free to open a case with TAC and add me to the case. |
|
Yes, it depends on the number of changes to be applied but on average, it completes around 2-3 changes each attempt and the rest fail. So if I increase the retries and play with the timeout values, I can get it to "complete" an apply. Obviously, this is completely unworkable. |
|
If you are saying it is only able to apply 2-3 changes, that is a bit surprising as I have been able to apply this (https://github.com/netascode/terraform-iosxe-evpn-example) config to four C9000v without any issues (134 resources in total) and without tuning any of the mentioned parameters. Which devices are you targeting? |
|
You may be missing the nuance that this is occuring when we make use of iosxe_template. I was able to do huge apply's when I'm not using the template. I had to use template because that was the only way we could specify some parameters in that version. |
|
Thank you, this was the missing link. |
|
The issue is related to the fact that certain config changes (e.g., applying a template to an interface) trigger a sync operation which locks the configuration database for a significant amount of time. Newer versions of XE might provider better results/performance. I have added an enhancement on the provider side that waits and blocks further operations for specific resources (ie. |
|
Tested with a batch of 12 ports on the latest provider release. Same result if I let it fail. Most of them iosxe_interface_ethernet.ports["2/0/13"]: Still modifying... [id=Cisco-IOS-XE-native:native/interface/GigabitEthernet=2%2F0%2F13, 2m30s elapsed] |
IOS-XE 17.9.5
We are using a for_each loop to create iosxe_interface_ethernet interfaces en masse. Each port is using an iosxe_template to supply the interface parameters.
The first creation succeeds. Other creates that are running in parallel fail with this lock error.
Error: Client Error
with iosxe_interface_ethernet.ports["4/0/32"],
on iosxe_interface_ethernet.ports.tf line 1, in resource "iosxe_interface_ethernet" "ports":
1: resource "iosxe_interface_ethernet" "ports" {
Failed to configure object (PATCH), got error: HTTP Request failed:
StatusCode 409, RESTCONF errors {Error:[{ErrorType:application
ErrorTag:lock-denied ErrorAppTag:
ErrorPath:/Cisco-IOS-XE-native:native/interface/GigabitEthernet
ErrorMessage:the configuration database is locked by session 18
yang_mgmt_infra tcp (system from 127.0.0.1) on since 2024-12-13 17:52:56
IOS-XE YANG Infrastructure ErrorInfo:}]} {PatchId: GlobalStatus:{Ok:false Errors:{Error:[]}} EditStatus:{Edit:[]} Errors:{Error:[]}}
The text was updated successfully, but these errors were encountered: