Skip to content

[action] [PR:21992] [dualtor] Fix mux start failure due to pre operation timeout #22027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 14, 2025

Conversation

mssonicbld
Copy link
Collaborator

Why I did it

Mux service is observed not started due to pre operation timeout:

2025 Mar  6 12:39:46.192880 vlab-06 NOTICE switch_hash: 'reload' executing with command: config reload -y
2025 Mar  6 12:40:47.490091 vlab-06 INFO systemd[1]: Starting swss.service - switch state service…
2025 Mar  6 12:40:57.156019 vlab-06 INFO systemd[1]: Starting mux.service - MUX Cable Container…
2025 Mar  6 12:41:22.474031 vlab-06 NOTICE swss#tunnelmgrd: :- doTunnelTask: Tunnel MuxTunnel0 task, op SET
2025 Mar  6 12:42:27.344856 vlab-06 WARNING systemd[1]: mux.service: start-pre operation timed out. Terminating.
2025 Mar  6 12:42:27.346562 vlab-06 WARNING systemd[1]: mux.service: Control process exited, code=killed, status=15/TERM
2025 Mar  6 12:43:11.903374 vlab-06 NOTICE swss#orchagent: :- doDecapTunnelTask: Tunnel MuxTunnel0 added to ASIC_DB.

Two issues:

  • The orchagent tunnel creation can takes up to 2mins after reload on kvm && mlnx platforms, so write_standby.py always timeout waiting for the tunnel to be ready.
  • The default pre operation timeout is only 90s.
Work item tracking
  • Microsoft ADO (number only): 31745242

How I did it

Let's increase the tunnel wait time in write_standby.py to 160s, and make the mux.service pre operation timeout as 180s.

How to verify it

Reload with mux running:

2025 Mar 11 08:07:38.865006 INFO systemd[1]: Starting mux.service - MUX Cable Container...
2025 Mar 11 08:09:52.938199 WARNING write_standby: Applying state to interfaces {'Ethernet0': 'active', 'Ethernet16': 'active', 'Ethernet160': 'active', 'Ethernet168': 'active', 'Ethernet176': 'active', 'Ethernet184': 'active', 'Ethernet192': 'active', 'Ethernet200': 'active', 'Ethernet208': 'active', 'Ethernet216': 'active', 'Ethernet224': 'active', 'Ethernet232': 'active', 'Ethernet24': 'active', 'Ethernet240': 'active', 'Ethernet32': 'active', 'Ethernet40': 'active', 'Ethernet48': 'active', 'Ethernet56': 'active', 'Ethernet64': 'active', 'Ethernet72': 'active', 'Ethernet8': 'active', 'Ethernet80': 'active', 'Ethernet88': 'active'}
2025 Mar 11 08:09:54.159939 INFO systemd[1]: Started mux.service - MUX Cable Container.

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211
  • 202305
  • 202411

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it
Mux service is observed not started due to pre operation timeout:
```
2025 Mar  6 12:39:46.192880 vlab-06 NOTICE switch_hash: 'reload' executing with command: config reload -y
2025 Mar  6 12:40:47.490091 vlab-06 INFO systemd[1]: Starting swss.service - switch state service…
2025 Mar  6 12:40:57.156019 vlab-06 INFO systemd[1]: Starting mux.service - MUX Cable Container…
2025 Mar  6 12:41:22.474031 vlab-06 NOTICE swss#tunnelmgrd: :- doTunnelTask: Tunnel MuxTunnel0 task, op SET
2025 Mar  6 12:42:27.344856 vlab-06 WARNING systemd[1]: mux.service: start-pre operation timed out. Terminating.
2025 Mar  6 12:42:27.346562 vlab-06 WARNING systemd[1]: mux.service: Control process exited, code=killed, status=15/TERM
2025 Mar  6 12:43:11.903374 vlab-06 NOTICE swss#orchagent: :- doDecapTunnelTask: Tunnel MuxTunnel0 added to ASIC_DB.
```
Two issues:
- The orchagent tunnel creation can takes up to 2mins after reload on kvm && mlnx platforms, so `write_standby.py` always timeout waiting for the tunnel to be ready.
- The default pre operation timeout is only 90s.

##### Work item tracking
- Microsoft ADO **(number only)**: 31745242

#### How I did it
Let's increase the tunnel wait time in `write_standby.py` to 160s, and make the `mux.service` pre operation timeout as 180s.

#### How to verify it
Reload with `mux` running:
```
2025 Mar 11 08:07:38.865006 INFO systemd[1]: Starting mux.service - MUX Cable Container...
2025 Mar 11 08:09:52.938199 WARNING write_standby: Applying state to interfaces {'Ethernet0': 'active', 'Ethernet16': 'active', 'Ethernet160': 'active', 'Ethernet168': 'active', 'Ethernet176': 'active', 'Ethernet184': 'active', 'Ethernet192': 'active', 'Ethernet200': 'active', 'Ethernet208': 'active', 'Ethernet216': 'active', 'Ethernet224': 'active', 'Ethernet232': 'active', 'Ethernet24': 'active', 'Ethernet240': 'active', 'Ethernet32': 'active', 'Ethernet40': 'active', 'Ethernet48': 'active', 'Ethernet56': 'active', 'Ethernet64': 'active', 'Ethernet72': 'active', 'Ethernet8': 'active', 'Ethernet80': 'active', 'Ethernet88': 'active'}
2025 Mar 11 08:09:54.159939 INFO systemd[1]: Started mux.service - MUX Cable Container.
```

<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012.
-->

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
- [x] 202411
#### Tested branch (Please provide the tested image version)

<!--
- Please provide tested image version
- e.g.
- [x] 20201231.100
-->

- [ ] <!-- image version 1 -->
- [ ] <!-- image version 2 -->

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md
-->

#### A picture of a cute animal (not mandatory but encouraged)
@mssonicbld
Copy link
Collaborator Author

Original PR: #21992

@mssonicbld
Copy link
Collaborator Author

/azp run Azure.sonic-buildimage

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@kperumalbfn
Copy link
Contributor

/azp run Azure.sonic-buildimage

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@kperumalbfn kperumalbfn enabled auto-merge (squash) March 14, 2025 05:07
@kperumalbfn kperumalbfn merged commit f88aa3a into sonic-net:202411 Mar 14, 2025
19 checks passed
dgsudharsan pushed a commit to dgsudharsan/sonic-buildimage that referenced this pull request Mar 17, 2025
…12 (sonic-net#893)

```<br>* cc750dbc9 - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-buildimage into 202412 (2025-03-15) [Sonic Automation]
* fa9c47b - (head/202411) Upgrade xgs SAI to 12.3.2.3 (sonic-net#22042) (2025-03-14) [zitingguo-ms]
* b7f0ca0 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22040) (2025-03-14) [mssonicbld]
* f88aa3a - [dualtor] Fix `mux` start failure due to pre operation timeout (sonic-net#22027) (2025-03-13) [mssonicbld]
* 49146b8 - [submodule] Update submodule sonic-sairedis to the latest HEAD automatically (sonic-net#22024) (2025-03-13) [mssonicbld]
* 54b0e3b - [202411] [Marvell-teralynx] Correction ISAI debian path (sonic-net#22032) (2025-03-13) [Keshav Gupta]
* 8a3dff1 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22025) (2025-03-13) [mssonicbld]
* 9072445 - [rsyslog]: Remote logging with the highest rule priority (sonic-net#22020) (2025-03-13) [mssonicbld]
* 3283497 - [Nokia-7215-A1] Update EZB files for SAI 15.1 (sonic-net#22004) (2025-03-13) [mssonicbld]
* c807b26 - Build fix for ptf-py3 issue on bullseye (sonic-net#21925) (2025-03-12) [mssonicbld]
* be28041 - Use pzstd to compress the Docker in SWI slim images (sonic-net#22005) (2025-03-12) [mssonicbld]
* c371cd3 - [202411]Upgrade SONiC package versions (sonic-net#21971) (2025-03-11) [mssonicbld]
* ad28af8 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21968) (2025-03-11) [mssonicbld]
* 6cb2941 - [YANG] Add yang model to `SUBNET_DECAP` table (sonic-net#21118) (sonic-net#21974) (2025-03-11) [Longxiang Lyu]
* ef9e911 - [Mellanox] Support new field "supporting_bulk_counter_groups" in DEVICE_METADATA|localhost for Mellanox-SN4280-O28 (sonic-net#21986) (2025-03-11) [mssonicbld]
* 39dbcfd - [SNMP] Fix config template issue when setting snmpagentaddress (sonic-net#21987) (2025-03-11) [mssonicbld]
* 5e6a202 - Merge pull request sonic-net#21964 from Aravind-Subbaroyan/patch-5 (2025-03-10) [Kumaresh Perumal]
|\ 
| failure_prs.log skip_prs.log e1be5dd - Update cisco-8000.ini (2025-03-07) [Aravind-Subbaroyan]
* | 37df4a3 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21912) (2025-03-08) [mssonicbld]
|/ 
* a414a7f - [submodule] Update submodule sonic-swss to the latest HEAD automatically (sonic-net#21933) (2025-03-08) [mssonicbld]<br>```
dgsudharsan pushed a commit to dgsudharsan/sonic-buildimage that referenced this pull request Mar 17, 2025
…12 (sonic-net#904)

```<br>* d17a92f8b - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-buildimage into 202412 (2025-03-16) [Sonic Automation]
* fa9c47b - (head/202411) Upgrade xgs SAI to 12.3.2.3 (sonic-net#22042) (2025-03-14) [zitingguo-ms]
* b7f0ca0 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22040) (2025-03-14) [mssonicbld]
* f88aa3a - [dualtor] Fix `mux` start failure due to pre operation timeout (sonic-net#22027) (2025-03-13) [mssonicbld]
* 49146b8 - [submodule] Update submodule sonic-sairedis to the latest HEAD automatically (sonic-net#22024) (2025-03-13) [mssonicbld]
* 54b0e3b - [202411] [Marvell-teralynx] Correction ISAI debian path (sonic-net#22032) (2025-03-13) [Keshav Gupta]
* 8a3dff1 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22025) (2025-03-13) [mssonicbld]
* 9072445 - [rsyslog]: Remote logging with the highest rule priority (sonic-net#22020) (2025-03-13) [mssonicbld]
* 3283497 - [Nokia-7215-A1] Update EZB files for SAI 15.1 (sonic-net#22004) (2025-03-13) [mssonicbld]
* c807b26 - Build fix for ptf-py3 issue on bullseye (sonic-net#21925) (2025-03-12) [mssonicbld]
* be28041 - Use pzstd to compress the Docker in SWI slim images (sonic-net#22005) (2025-03-12) [mssonicbld]
* c371cd3 - [202411]Upgrade SONiC package versions (sonic-net#21971) (2025-03-11) [mssonicbld]
* ad28af8 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21968) (2025-03-11) [mssonicbld]
* 6cb2941 - [YANG] Add yang model to `SUBNET_DECAP` table (sonic-net#21118) (sonic-net#21974) (2025-03-11) [Longxiang Lyu]
* ef9e911 - [Mellanox] Support new field "supporting_bulk_counter_groups" in DEVICE_METADATA|localhost for Mellanox-SN4280-O28 (sonic-net#21986) (2025-03-11) [mssonicbld]
* 39dbcfd - [SNMP] Fix config template issue when setting snmpagentaddress (sonic-net#21987) (2025-03-11) [mssonicbld]
* 5e6a202 - Merge pull request sonic-net#21964 from Aravind-Subbaroyan/patch-5 (2025-03-10) [Kumaresh Perumal]
|\ 
| failure_prs.log skip_prs.log e1be5dd - Update cisco-8000.ini (2025-03-07) [Aravind-Subbaroyan]
* | 37df4a3 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21912) (2025-03-08) [mssonicbld]
|/ 
* a414a7f - [submodule] Update submodule sonic-swss to the latest HEAD automatically (sonic-net#21933) (2025-03-08) [mssonicbld]<br>```
dgsudharsan pushed a commit to dgsudharsan/sonic-buildimage that referenced this pull request Mar 17, 2025
…12 (sonic-net#913)

```<br>* fb574259f - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-buildimage into 202412 (2025-03-17) [Sonic Automation]
* a85d973 - (head/202411) [Arista] Change port layout for Arista-7050CX3-32S-C28S4 (sonic-net#22003) (2025-03-16) [mssonicbld]
* fa9c47b - Upgrade xgs SAI to 12.3.2.3 (sonic-net#22042) (2025-03-14) [zitingguo-ms]
* b7f0ca0 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22040) (2025-03-14) [mssonicbld]
* f88aa3a - [dualtor] Fix `mux` start failure due to pre operation timeout (sonic-net#22027) (2025-03-13) [mssonicbld]
* 49146b8 - [submodule] Update submodule sonic-sairedis to the latest HEAD automatically (sonic-net#22024) (2025-03-13) [mssonicbld]
* 54b0e3b - [202411] [Marvell-teralynx] Correction ISAI debian path (sonic-net#22032) (2025-03-13) [Keshav Gupta]
* 8a3dff1 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22025) (2025-03-13) [mssonicbld]
* 9072445 - [rsyslog]: Remote logging with the highest rule priority (sonic-net#22020) (2025-03-13) [mssonicbld]
* 3283497 - [Nokia-7215-A1] Update EZB files for SAI 15.1 (sonic-net#22004) (2025-03-13) [mssonicbld]
* c807b26 - Build fix for ptf-py3 issue on bullseye (sonic-net#21925) (2025-03-12) [mssonicbld]
* be28041 - Use pzstd to compress the Docker in SWI slim images (sonic-net#22005) (2025-03-12) [mssonicbld]
* c371cd3 - [202411]Upgrade SONiC package versions (sonic-net#21971) (2025-03-11) [mssonicbld]
* ad28af8 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21968) (2025-03-11) [mssonicbld]
* 6cb2941 - [YANG] Add yang model to `SUBNET_DECAP` table (sonic-net#21118) (sonic-net#21974) (2025-03-11) [Longxiang Lyu]
* ef9e911 - [Mellanox] Support new field "supporting_bulk_counter_groups" in DEVICE_METADATA|localhost for Mellanox-SN4280-O28 (sonic-net#21986) (2025-03-11) [mssonicbld]
* 39dbcfd - [SNMP] Fix config template issue when setting snmpagentaddress (sonic-net#21987) (2025-03-11) [mssonicbld]
* 5e6a202 - Merge pull request sonic-net#21964 from Aravind-Subbaroyan/patch-5 (2025-03-10) [Kumaresh Perumal]
|\ 
| failure_prs.log skip_prs.log e1be5dd - Update cisco-8000.ini (2025-03-07) [Aravind-Subbaroyan]
* | 37df4a3 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21912) (2025-03-08) [mssonicbld]
|/ 
* a414a7f - [submodule] Update submodule sonic-swss to the latest HEAD automatically (sonic-net#21933) (2025-03-08) [mssonicbld]<br>```
vivekrnv pushed a commit to vivekrnv/sonic-buildimage that referenced this pull request Mar 24, 2025
```<br>* aa24d5a - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-buildimage into 202412 (2025-03-18) [Sonic Automation]
* bdc62d6 - (head/202411) Update fips openssh to openssh_9.2p1-2+deb12u5 (sonic-net#22028) (2025-03-18) [mssonicbld]
* 89f3bdc - [Mellanox] Remove SKU ACS-SN4280 (sonic-net#22035) (2025-03-18) [mssonicbld]
* a85d973 - [Arista] Change port layout for Arista-7050CX3-32S-C28S4 (sonic-net#22003) (2025-03-16) [mssonicbld]
* fa9c47b - Upgrade xgs SAI to 12.3.2.3 (sonic-net#22042) (2025-03-14) [zitingguo-ms]
* b7f0ca0 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22040) (2025-03-14) [mssonicbld]
* f88aa3a - [dualtor] Fix `mux` start failure due to pre operation timeout (sonic-net#22027) (2025-03-13) [mssonicbld]
* 49146b8 - [submodule] Update submodule sonic-sairedis to the latest HEAD automatically (sonic-net#22024) (2025-03-13) [mssonicbld]
* 54b0e3b - [202411] [Marvell-teralynx] Correction ISAI debian path (sonic-net#22032) (2025-03-13) [Keshav Gupta]
* 8a3dff1 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#22025) (2025-03-13) [mssonicbld]
* 9072445 - [rsyslog]: Remote logging with the highest rule priority (sonic-net#22020) (2025-03-13) [mssonicbld]
* 3283497 - [Nokia-7215-A1] Update EZB files for SAI 15.1 (sonic-net#22004) (2025-03-13) [mssonicbld]
* c807b26 - Build fix for ptf-py3 issue on bullseye (sonic-net#21925) (2025-03-12) [mssonicbld]
* be28041 - Use pzstd to compress the Docker in SWI slim images (sonic-net#22005) (2025-03-12) [mssonicbld]
* c371cd3 - [202411]Upgrade SONiC package versions (sonic-net#21971) (2025-03-11) [mssonicbld]
* ad28af8 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21968) (2025-03-11) [mssonicbld]
* 6cb2941 - [YANG] Add yang model to `SUBNET_DECAP` table (sonic-net#21118) (sonic-net#21974) (2025-03-11) [Longxiang Lyu]
* ef9e911 - [Mellanox] Support new field "supporting_bulk_counter_groups" in DEVICE_METADATA|localhost for Mellanox-SN4280-O28 (sonic-net#21986) (2025-03-11) [mssonicbld]
* 39dbcfd - [SNMP] Fix config template issue when setting snmpagentaddress (sonic-net#21987) (2025-03-11) [mssonicbld]
* 5e6a202 - Merge pull request sonic-net#21964 from Aravind-Subbaroyan/patch-5 (2025-03-10) [Kumaresh Perumal]
|\ 
| failure_prs.log skip_prs.log e1be5dd - Update cisco-8000.ini (2025-03-07) [Aravind-Subbaroyan]
* | 37df4a3 - [submodule] Update submodule sonic-utilities to the latest HEAD automatically (sonic-net#21912) (2025-03-08) [mssonicbld]
|/ 
* a414a7f - [submodule] Update submodule sonic-swss to the latest HEAD automatically (sonic-net#21933) (2025-03-08) [mssonicbld]<br>```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants