-
Notifications
You must be signed in to change notification settings - Fork 1.5k
[action] [PR:21992] [dualtor] Fix mux
start failure due to pre operation timeout
#22027
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
<!-- Please make sure you've read and understood our contributing guidelines: https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md ** Make sure all your commits include a signature generated with `git commit -s` ** If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx" or "resolves #xxxx" Please provide the following information: --> #### Why I did it Mux service is observed not started due to pre operation timeout: ``` 2025 Mar 6 12:39:46.192880 vlab-06 NOTICE switch_hash: 'reload' executing with command: config reload -y 2025 Mar 6 12:40:47.490091 vlab-06 INFO systemd[1]: Starting swss.service - switch state service… 2025 Mar 6 12:40:57.156019 vlab-06 INFO systemd[1]: Starting mux.service - MUX Cable Container… 2025 Mar 6 12:41:22.474031 vlab-06 NOTICE swss#tunnelmgrd: :- doTunnelTask: Tunnel MuxTunnel0 task, op SET 2025 Mar 6 12:42:27.344856 vlab-06 WARNING systemd[1]: mux.service: start-pre operation timed out. Terminating. 2025 Mar 6 12:42:27.346562 vlab-06 WARNING systemd[1]: mux.service: Control process exited, code=killed, status=15/TERM 2025 Mar 6 12:43:11.903374 vlab-06 NOTICE swss#orchagent: :- doDecapTunnelTask: Tunnel MuxTunnel0 added to ASIC_DB. ``` Two issues: - The orchagent tunnel creation can takes up to 2mins after reload on kvm && mlnx platforms, so `write_standby.py` always timeout waiting for the tunnel to be ready. - The default pre operation timeout is only 90s. ##### Work item tracking - Microsoft ADO **(number only)**: 31745242 #### How I did it Let's increase the tunnel wait time in `write_standby.py` to 160s, and make the `mux.service` pre operation timeout as 180s. #### How to verify it Reload with `mux` running: ``` 2025 Mar 11 08:07:38.865006 INFO systemd[1]: Starting mux.service - MUX Cable Container... 2025 Mar 11 08:09:52.938199 WARNING write_standby: Applying state to interfaces {'Ethernet0': 'active', 'Ethernet16': 'active', 'Ethernet160': 'active', 'Ethernet168': 'active', 'Ethernet176': 'active', 'Ethernet184': 'active', 'Ethernet192': 'active', 'Ethernet200': 'active', 'Ethernet208': 'active', 'Ethernet216': 'active', 'Ethernet224': 'active', 'Ethernet232': 'active', 'Ethernet24': 'active', 'Ethernet240': 'active', 'Ethernet32': 'active', 'Ethernet40': 'active', 'Ethernet48': 'active', 'Ethernet56': 'active', 'Ethernet64': 'active', 'Ethernet72': 'active', 'Ethernet8': 'active', 'Ethernet80': 'active', 'Ethernet88': 'active'} 2025 Mar 11 08:09:54.159939 INFO systemd[1]: Started mux.service - MUX Cable Container. ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [ ] 202205 - [ ] 202211 - [ ] 202305 - [x] 202411 #### Tested branch (Please provide the tested image version) <!-- - Please provide tested image version - e.g. - [x] 20201231.100 --> - [ ] <!-- image version 1 --> - [ ] <!-- image version 2 --> #### Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> <!-- Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU. --> #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged)
Original PR: #21992 |
12 tasks
/azp run Azure.sonic-buildimage |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run Azure.sonic-buildimage |
Azure Pipelines successfully started running 1 pipeline(s). |
kperumalbfn
approved these changes
Mar 14, 2025
dgsudharsan
pushed a commit
to dgsudharsan/sonic-buildimage
that referenced
this pull request
Mar 17, 2025
…12 (sonic-net#893) ```<br>* cc750dbc9 - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-buildimage into 202412 (2025-03-15) [Sonic Automation] * fa9c47b - (head/202411) Upgrade xgs SAI to 12.3.2.3 (sonic-net#22042) (2025-03-14) [zitingguo-ms] * b7f0ca0 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22040) (2025-03-14) [mssonicbld] * f88aa3a - [dualtor] Fix `mux` start failure due to pre operation timeout (sonic-net#22027) (2025-03-13) [mssonicbld] * 49146b8 - [submodule] Update submodule sonic-sairedis to the latest HEAD automatically (sonic-net#22024) (2025-03-13) [mssonicbld] * 54b0e3b - [202411] [Marvell-teralynx] Correction ISAI debian path (sonic-net#22032) (2025-03-13) [Keshav Gupta] * 8a3dff1 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22025) (2025-03-13) [mssonicbld] * 9072445 - [rsyslog]: Remote logging with the highest rule priority (sonic-net#22020) (2025-03-13) [mssonicbld] * 3283497 - [Nokia-7215-A1] Update EZB files for SAI 15.1 (sonic-net#22004) (2025-03-13) [mssonicbld] * c807b26 - Build fix for ptf-py3 issue on bullseye (sonic-net#21925) (2025-03-12) [mssonicbld] * be28041 - Use pzstd to compress the Docker in SWI slim images (sonic-net#22005) (2025-03-12) [mssonicbld] * c371cd3 - [202411]Upgrade SONiC package versions (sonic-net#21971) (2025-03-11) [mssonicbld] * ad28af8 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21968) (2025-03-11) [mssonicbld] * 6cb2941 - [YANG] Add yang model to `SUBNET_DECAP` table (sonic-net#21118) (sonic-net#21974) (2025-03-11) [Longxiang Lyu] * ef9e911 - [Mellanox] Support new field "supporting_bulk_counter_groups" in DEVICE_METADATA|localhost for Mellanox-SN4280-O28 (sonic-net#21986) (2025-03-11) [mssonicbld] * 39dbcfd - [SNMP] Fix config template issue when setting snmpagentaddress (sonic-net#21987) (2025-03-11) [mssonicbld] * 5e6a202 - Merge pull request sonic-net#21964 from Aravind-Subbaroyan/patch-5 (2025-03-10) [Kumaresh Perumal] |\ | failure_prs.log skip_prs.log e1be5dd - Update cisco-8000.ini (2025-03-07) [Aravind-Subbaroyan] * | 37df4a3 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21912) (2025-03-08) [mssonicbld] |/ * a414a7f - [submodule] Update submodule sonic-swss to the latest HEAD automatically (sonic-net#21933) (2025-03-08) [mssonicbld]<br>```
dgsudharsan
pushed a commit
to dgsudharsan/sonic-buildimage
that referenced
this pull request
Mar 17, 2025
…12 (sonic-net#904) ```<br>* d17a92f8b - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-buildimage into 202412 (2025-03-16) [Sonic Automation] * fa9c47b - (head/202411) Upgrade xgs SAI to 12.3.2.3 (sonic-net#22042) (2025-03-14) [zitingguo-ms] * b7f0ca0 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22040) (2025-03-14) [mssonicbld] * f88aa3a - [dualtor] Fix `mux` start failure due to pre operation timeout (sonic-net#22027) (2025-03-13) [mssonicbld] * 49146b8 - [submodule] Update submodule sonic-sairedis to the latest HEAD automatically (sonic-net#22024) (2025-03-13) [mssonicbld] * 54b0e3b - [202411] [Marvell-teralynx] Correction ISAI debian path (sonic-net#22032) (2025-03-13) [Keshav Gupta] * 8a3dff1 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22025) (2025-03-13) [mssonicbld] * 9072445 - [rsyslog]: Remote logging with the highest rule priority (sonic-net#22020) (2025-03-13) [mssonicbld] * 3283497 - [Nokia-7215-A1] Update EZB files for SAI 15.1 (sonic-net#22004) (2025-03-13) [mssonicbld] * c807b26 - Build fix for ptf-py3 issue on bullseye (sonic-net#21925) (2025-03-12) [mssonicbld] * be28041 - Use pzstd to compress the Docker in SWI slim images (sonic-net#22005) (2025-03-12) [mssonicbld] * c371cd3 - [202411]Upgrade SONiC package versions (sonic-net#21971) (2025-03-11) [mssonicbld] * ad28af8 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21968) (2025-03-11) [mssonicbld] * 6cb2941 - [YANG] Add yang model to `SUBNET_DECAP` table (sonic-net#21118) (sonic-net#21974) (2025-03-11) [Longxiang Lyu] * ef9e911 - [Mellanox] Support new field "supporting_bulk_counter_groups" in DEVICE_METADATA|localhost for Mellanox-SN4280-O28 (sonic-net#21986) (2025-03-11) [mssonicbld] * 39dbcfd - [SNMP] Fix config template issue when setting snmpagentaddress (sonic-net#21987) (2025-03-11) [mssonicbld] * 5e6a202 - Merge pull request sonic-net#21964 from Aravind-Subbaroyan/patch-5 (2025-03-10) [Kumaresh Perumal] |\ | failure_prs.log skip_prs.log e1be5dd - Update cisco-8000.ini (2025-03-07) [Aravind-Subbaroyan] * | 37df4a3 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21912) (2025-03-08) [mssonicbld] |/ * a414a7f - [submodule] Update submodule sonic-swss to the latest HEAD automatically (sonic-net#21933) (2025-03-08) [mssonicbld]<br>```
dgsudharsan
pushed a commit
to dgsudharsan/sonic-buildimage
that referenced
this pull request
Mar 17, 2025
…12 (sonic-net#913) ```<br>* fb574259f - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-buildimage into 202412 (2025-03-17) [Sonic Automation] * a85d973 - (head/202411) [Arista] Change port layout for Arista-7050CX3-32S-C28S4 (sonic-net#22003) (2025-03-16) [mssonicbld] * fa9c47b - Upgrade xgs SAI to 12.3.2.3 (sonic-net#22042) (2025-03-14) [zitingguo-ms] * b7f0ca0 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22040) (2025-03-14) [mssonicbld] * f88aa3a - [dualtor] Fix `mux` start failure due to pre operation timeout (sonic-net#22027) (2025-03-13) [mssonicbld] * 49146b8 - [submodule] Update submodule sonic-sairedis to the latest HEAD automatically (sonic-net#22024) (2025-03-13) [mssonicbld] * 54b0e3b - [202411] [Marvell-teralynx] Correction ISAI debian path (sonic-net#22032) (2025-03-13) [Keshav Gupta] * 8a3dff1 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22025) (2025-03-13) [mssonicbld] * 9072445 - [rsyslog]: Remote logging with the highest rule priority (sonic-net#22020) (2025-03-13) [mssonicbld] * 3283497 - [Nokia-7215-A1] Update EZB files for SAI 15.1 (sonic-net#22004) (2025-03-13) [mssonicbld] * c807b26 - Build fix for ptf-py3 issue on bullseye (sonic-net#21925) (2025-03-12) [mssonicbld] * be28041 - Use pzstd to compress the Docker in SWI slim images (sonic-net#22005) (2025-03-12) [mssonicbld] * c371cd3 - [202411]Upgrade SONiC package versions (sonic-net#21971) (2025-03-11) [mssonicbld] * ad28af8 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21968) (2025-03-11) [mssonicbld] * 6cb2941 - [YANG] Add yang model to `SUBNET_DECAP` table (sonic-net#21118) (sonic-net#21974) (2025-03-11) [Longxiang Lyu] * ef9e911 - [Mellanox] Support new field "supporting_bulk_counter_groups" in DEVICE_METADATA|localhost for Mellanox-SN4280-O28 (sonic-net#21986) (2025-03-11) [mssonicbld] * 39dbcfd - [SNMP] Fix config template issue when setting snmpagentaddress (sonic-net#21987) (2025-03-11) [mssonicbld] * 5e6a202 - Merge pull request sonic-net#21964 from Aravind-Subbaroyan/patch-5 (2025-03-10) [Kumaresh Perumal] |\ | failure_prs.log skip_prs.log e1be5dd - Update cisco-8000.ini (2025-03-07) [Aravind-Subbaroyan] * | 37df4a3 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21912) (2025-03-08) [mssonicbld] |/ * a414a7f - [submodule] Update submodule sonic-swss to the latest HEAD automatically (sonic-net#21933) (2025-03-08) [mssonicbld]<br>```
vivekrnv
pushed a commit
to vivekrnv/sonic-buildimage
that referenced
this pull request
Mar 24, 2025
```<br>* aa24d5a - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-buildimage into 202412 (2025-03-18) [Sonic Automation] * bdc62d6 - (head/202411) Update fips openssh to openssh_9.2p1-2+deb12u5 (sonic-net#22028) (2025-03-18) [mssonicbld] * 89f3bdc - [Mellanox] Remove SKU ACS-SN4280 (sonic-net#22035) (2025-03-18) [mssonicbld] * a85d973 - [Arista] Change port layout for Arista-7050CX3-32S-C28S4 (sonic-net#22003) (2025-03-16) [mssonicbld] * fa9c47b - Upgrade xgs SAI to 12.3.2.3 (sonic-net#22042) (2025-03-14) [zitingguo-ms] * b7f0ca0 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22040) (2025-03-14) [mssonicbld] * f88aa3a - [dualtor] Fix `mux` start failure due to pre operation timeout (sonic-net#22027) (2025-03-13) [mssonicbld] * 49146b8 - [submodule] Update submodule sonic-sairedis to the latest HEAD automatically (sonic-net#22024) (2025-03-13) [mssonicbld] * 54b0e3b - [202411] [Marvell-teralynx] Correction ISAI debian path (sonic-net#22032) (2025-03-13) [Keshav Gupta] * 8a3dff1 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22025) (2025-03-13) [mssonicbld] * 9072445 - [rsyslog]: Remote logging with the highest rule priority (sonic-net#22020) (2025-03-13) [mssonicbld] * 3283497 - [Nokia-7215-A1] Update EZB files for SAI 15.1 (sonic-net#22004) (2025-03-13) [mssonicbld] * c807b26 - Build fix for ptf-py3 issue on bullseye (sonic-net#21925) (2025-03-12) [mssonicbld] * be28041 - Use pzstd to compress the Docker in SWI slim images (sonic-net#22005) (2025-03-12) [mssonicbld] * c371cd3 - [202411]Upgrade SONiC package versions (sonic-net#21971) (2025-03-11) [mssonicbld] * ad28af8 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21968) (2025-03-11) [mssonicbld] * 6cb2941 - [YANG] Add yang model to `SUBNET_DECAP` table (sonic-net#21118) (sonic-net#21974) (2025-03-11) [Longxiang Lyu] * ef9e911 - [Mellanox] Support new field "supporting_bulk_counter_groups" in DEVICE_METADATA|localhost for Mellanox-SN4280-O28 (sonic-net#21986) (2025-03-11) [mssonicbld] * 39dbcfd - [SNMP] Fix config template issue when setting snmpagentaddress (sonic-net#21987) (2025-03-11) [mssonicbld] * 5e6a202 - Merge pull request sonic-net#21964 from Aravind-Subbaroyan/patch-5 (2025-03-10) [Kumaresh Perumal] |\ | failure_prs.log skip_prs.log e1be5dd - Update cisco-8000.ini (2025-03-07) [Aravind-Subbaroyan] * | 37df4a3 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21912) (2025-03-08) [mssonicbld] |/ * a414a7f - [submodule] Update submodule sonic-swss to the latest HEAD automatically (sonic-net#21933) (2025-03-08) [mssonicbld]<br>```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I did it
Mux service is observed not started due to pre operation timeout:
Two issues:
write_standby.py
always timeout waiting for the tunnel to be ready.Work item tracking
How I did it
Let's increase the tunnel wait time in
write_standby.py
to 160s, and make themux.service
pre operation timeout as 180s.How to verify it
Reload with
mux
running:Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)