Do not enter vendor SAI critical section for counter polling/clearing operations #1450
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Stephen Sun <[email protected]>
Signed-off-by: Stephen Sun <[email protected]>
This reverts commit 32ed1c2.
This reverts commit 8d2aa54.
… community maintainer Signed-off-by: Stephen Sun <[email protected]>
Junchao-Mellanox
approved these changes
Nov 4, 2024
kcudnik
approved these changes
Nov 4, 2024
shiraez
pushed a commit
to Marvell-switching/sonic-sairedis
that referenced
this pull request
Dec 12, 2024
… operations (sonic-net#1450) There are two types of threads in syncd: The syncd main thread in which the create/destroy/set/get SAI APIs are called Flex counter threads in which counter-polling/-clearing APIs are called The critical section in vendor SAI was introduced to protect vendors' SAI from being re-entered, which prevents the flex counter threads from running concurrently, and introduces latency for one flex counter thread when it's waiting for the critical section. It is not necessary to enter a section in counter-polling/clearing operations since the objects' state won't be changed in that API.
shiraez
pushed a commit
to Marvell-switching/sonic-sairedis
that referenced
this pull request
Dec 12, 2024
… operations (sonic-net#1450) There are two types of threads in syncd: The syncd main thread in which the create/destroy/set/get SAI APIs are called Flex counter threads in which counter-polling/-clearing APIs are called The critical section in vendor SAI was introduced to protect vendors' SAI from being re-entered, which prevents the flex counter threads from running concurrently, and introduces latency for one flex counter thread when it's waiting for the critical section. It is not necessary to enter a section in counter-polling/clearing operations since the objects' state won't be changed in that API.
|
@stephenxs @kcudnik I would not include this change in 202405 branch as we are finishing the qualification for 202405. Please let me know if you think this is a must-have change for 202405. |
|
I'm not able to answer that question, that change was made to fix counter pooling fix, I think it would b3 nic to have at least, please ask @stephenxs |
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Jan 12, 2025
…clearing operations (sonic-net#1450)" This reverts commit 0317b16.
|
@stephenxs Removing this PR from 202411 for now. Please confirm the resolution of the timeout issue you mentioned in the revert PR. |
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Jan 16, 2025
…clearing operations (sonic-net#1450)" This reverts commit 0317b16.
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Jan 20, 2025
…clearing operations (sonic-net#1450)" This reverts commit 0317b16.
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Jan 30, 2025
…clearing operations (sonic-net#1450)" This reverts commit 0317b16.
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Feb 4, 2025
…clearing operations (sonic-net#1450)" This reverts commit 0317b16.
stephenxs
added a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Feb 7, 2025
…clearing operations (sonic-net#1450)" This reverts commit 0317b16.
kcudnik
pushed a commit
that referenced
this pull request
Feb 10, 2025
|
@stephenxs could you please fix the cherrypick conflict in 202411 |
Thanks @kperumalbfn |
rpmarvell
pushed a commit
to rpmarvell/sonic-sairedis
that referenced
this pull request
Feb 13, 2025
…clearing operations (sonic-net#1450)" (sonic-net#1498) Revert "Do not enter vendor SAI critical section for counter polling/clearing operations (sonic-net#1450)" This reverts commit 0317b16.
|
@kperumalbfn could you check whether this PR is in 202411? it is reverted from master and you said it had been removed in 202411. but I still see it in 202411. |
|
@kperumalbfn I checked 202411 branch and confirmed this change is in 202411 branch because PR was merged before 202411 got cut off. So we need to cherry-pick the revert PR #1450 |
mssonicbld
added a commit
to mssonicbld/sonic-sairedis
that referenced
this pull request
Feb 26, 2025
…clearing operations (sonic-net#1450)" Revert "Do not enter vendor SAI critical section for counter polling/clearing operations (sonic-net#1450)" This reverts commit 0317b16.
mssonicbld
added a commit
that referenced
this pull request
Feb 26, 2025
stephenxs
pushed a commit
to stephenxs/sonic-sairedis
that referenced
this pull request
Feb 27, 2025
…#24) ```<br>* f83634d9 - (HEAD -> 202412) Merge branch '202411' of https://github.com/sonic-net/sonic-sairedis into 202412 (2025-02-27) [Sonic Automation] * 8c17d4b - (origin/202411) Revert "Do not enter vendor SAI critical section for counter polling/clearing operations (sonic-net#1450)" (sonic-net#1541) (2025-02-27) [mssonicbld] * 3df03e1 - Optimize counter polling interval by making it more accurate (sonic-net#1457) (sonic-net#1534) (2025-02-26) [Stephen Sun] * d884ff9 - [syncd] Move logSet logGet under mutex to prevent race condition (sonic-net#1520) (sonic-net#1538) (2025-02-20) [Kamil Cudnik] * ec8b3c3 - Fix pipeline errors related to rsyslogd and libswsscommon installation (sonic-net#1535) (2025-02-19) [mssonicbld] * 6b263b8 - [FC] Support Policer Counter (sonic-net#1533) (2025-02-19) [mssonicbld] * e53489e - [syncd] Update log level for bulk api (sonic-net#1532) (2025-02-18) [Jianyue Wu] * 7ae00e5 - Define bulk chunk size and bulk chunk size per counter ID (sonic-net#1528) (2025-02-11) [mssonicbld] * f35e743 - [nvidia] Skip SAI discovery on ports (sonic-net#1524) (2025-02-07) [mssonicbld] * bf049ed - Use sonictest pool instead of sonic-common and fix arm64 issue. (sonic-net#1516) (2025-02-05) [mssonicbld] * ffe371d - [syncd] Support bulk set in INIT_VIEW mode (sonic-net#1517) (2025-02-05) [mssonicbld]<br>```
stepanblyschak
pushed a commit
to stepanblyschak/sonic-sairedis
that referenced
this pull request
Feb 27, 2025
…clearing operations (sonic-net#1450)" This reverts commit 0317b16.
DavidZagury
pushed a commit
to DavidZagury/sonic-sairedis
that referenced
this pull request
Apr 28, 2025
* [syncd] Support bulk set in INIT_VIEW mode (sonic-net#1517) Support bulk set in INIT_VIEW mode. * Use sonictest pool instead of sonic-common and fix arm64 issue. (sonic-net#1516) 1. Use sonictest pool instead of sonic-common 2. Fix arm64 build error. * [nvidia] Skip SAI discovery on ports (sonic-net#1524) Given that modern systems have lots of ports, performing SAI discovery takes very long time, e.g. (8 sec) for 256 port system. This has a big impact of fast-boot downtime and the discovery itself is not required for Nvidia platform fast-boot. Same applies to Nvidia fastfast-boot (aka warm-boot), yet needs to be tested separately. * Define bulk chunk size and bulk chunk size per counter ID (sonic-net#1528) Define bulk chunk size and bulk chunk size per counter ID. This is to resolve the VS test failure in sonic-net#1457, which is caused by loop dependency. In PR sonic-net#1457, new fields `bulk_chunk_size` and `bulk_chunk_size_per_prefix` have been introduced to `sai_redis_flex_counter_group_parameter_t` whose instances are initialized by orchagent. However, the orchagent is still compiled with the old sairedis header, which prevents both new fields from being uninitialized which in turn fails vs test. We have to split this PR into two: 1. sonic-net#1519 which updates the header sairedis.h only. the motivation is to compile swss(orchagent) with both new fields initiated. 2. sonic-net#1457 contains all the rest of code The order to merge: 1. sonic-net#1519 2. sonic-net/sonic-swss#3391 3. sonic-net#1457 * [syncd] Update log level for bulk api (sonic-net#1532) [syncd] Update log level for bulk api * [FC] Support Policer Counter (sonic-net#1533) Added the implantation for policer counter - Support in POLICER group and sai_serialize functions Unit Tests: Included unit tests to add and remove policer counter. * Fix pipeline errors related to rsyslogd and libswsscommon installation (sonic-net#1535) On arm64 (and maybe sometimes amd64), rsyslogd appears to need a second or two to actually fully exit. The current code expects it to exit practically instantly. Add a sleep of 2 seconds to give it some time. Also enable some logging so that the commands being run can be seen. Also, fix an error related to libswsscommon not getting installed due to new dependencies being added. Solve this by using apt install to install the package, which brings in any necessary dependencies. * [syncd] Move logSet logGet under mutex to prevent race condition (sonic-net#1520) (sonic-net#1538) [syncd] Move logSet logGet under mutex to prevent race condition * Optimize counter polling interval by making it more accurate (sonic-net#1457) (sonic-net#1534) What I did Optimize the counter-polling performance in terms of polling interval accuracy Enable bulk counter-polling to run at a smaller chunk size There is one counter-polling thread for each counter group. All such threads can compete for the critical sections at the vendor SAI level, which means a counter-polling thread can wait for a critical section if another thread has been in it, which introduces latency for the waiting counter group. An example is the competition between the PFC watchdog and the port counter groups. The port counter group contains many counters and is polled in a bulk mode which takes a relatively longer time. The PFC watchdog counter group contains only a few counters but is polled at a short interval. Sometimes, PFC watchdog counters need to wait before polling, which makes the polling interval inaccurate and prevents the PFC storm from being detected in time. To resolve this issue, we can reduce the chunk size of the port counter group. The port counter group polls the counters of all ports in a single bulk operation by default. By using a smaller chunk size, it polls the counters in several bulk operations with each polling counter of a subset (whose size <= chunk size) of all ports. By doing so, the port counter group stays in the critical section for a shorter time and the PFC watchdog is more likely to be scheduled to poll counters and detect the PFC storm in time. Collect the time stamp immediately after vendor SAI API returns. Currently, many counter groups require a Lua plugin to execute based on polling interval, to calculate rates, detect certain events, etc. Eg. For PFC watchdog counter group to PFC storm. In this case, the polling interval is calculated based on the difference of time stamps between the current and last poll to avoid deviation due to scheduling latency. However, the timestamp is collected in the Lua plugin which is several steps after the SAI API returns and is executed in a different context (redis-server). Both introduce even larger deviations. To overcome this, we collect the timestamp immediately after the SAI API returns. * Revert "Do not enter vendor SAI critical section for counter polling/clearing operations (sonic-net#1450)" (sonic-net#1541) Revert "Do not enter vendor SAI critical section for counter polling/clearing operations (sonic-net#1450)" This reverts commit 0317b16. * [vslib] SAI_KEY_VS_OPER_SPEED_IS_CONFIGURED_SPEED, SAI_PORT_ATTR_HOST_TX_READY_STATUS support (sonic-net#1553) This PR adds two features to `vslib`. - `SAI_KEY_VS_OPER_SPEED_IS_CONFIGURED_SPEED`: when `true`, `SAI_PORT_ATTR_SPEED` returns the configured speed instead of the value retrieved via [`/sys/class/net/<name>/speed`](https://github.com/sonic-net/sonic-sairedis/blob/master/vslib/SwitchStateBaseHostif.cpp#L892-L893). - fixes sonic-net/sonic-buildimage#19735 - `SAI_PORT_ATTR_HOST_TX_READY_STATUS`: always returns `true`. Required to support running `xcvrd` in the VS env. - ref: https://github.com/sonic-net/SONiC/pull/1849/files#diff-6f3e95e6c57a3edc2e30e1f13edb9fd9a32a0db44e1035ac1f0b1b9a191762a5R46 * Update build_and_install_module.sh to match newer Linux kernel version (sonic-net#1561) sonic-sairedis will checkout sonic-swss to do vstest but using local build_and_install_module.sh to setup test environment, which is out of date with newer Linux kernel version. The build_and_install_module.sh in sonic-swss is up to date with latest Ubuntu 20.04, so we need to update the build sh file with the file in sonic-swss. In a long term, we may need to do some automatically sync, but now we have some azure agent security issue need to fix immediately, so just update the build_and_install_module.sh manually. * Revert "Optimize counter polling interval by making it more accurate (sonic-net#1457) …" (sonic-net#1570) Revert "Optimize counter polling interval by making it more accurate --------- Co-authored-by: mssonicbld <[email protected]> Co-authored-by: Jianyue Wu <[email protected]> Co-authored-by: Kamil Cudnik <[email protected]> Co-authored-by: Stephen Sun <[email protected]> Co-authored-by: Kumaresh Perumal <[email protected]>
DavidZagury
pushed a commit
to DavidZagury/sonic-sairedis
that referenced
this pull request
Apr 28, 2025
Code sync sonic-net/sonic-sairedis:202411 => 202412 ``` * 0fcc968 (HEAD -> code-sync-202412, origin/code-sync-202412) r12f 250417:0536 - Merge remote-tracking branch 'base/202411' into code-sync-202412 |\ | * 4048483 (base/202411) Kumaresh Perumal 250403:0905 - Revert "Optimize counter polling interval by making it more accurate (sonic-net#1457) …" (sonic-net#1570) | * 420d92f (user/riffjiang/last-202411) mssonicbld 250319:1302 - Update build_and_install_module.sh to match newer Linux kernel version (sonic-net#1561) | * e2d2ca6 mssonicbld 250312:1602 - [vslib] SAI_KEY_VS_OPER_SPEED_IS_CONFIGURED_SPEED, SAI_PORT_ATTR_HOST_TX_READY_STATUS support (sonic-net#1553) | * 8c17d4b mssonicbld 250227:0702 - Revert "Do not enter vendor SAI critical section for counter polling/clearing operations (sonic-net#1450)" (sonic-net#1541) | * 3df03e1 Stephen Sun 250226:1737 - Optimize counter polling interval by making it more accurate (sonic-net#1457) (sonic-net#1534) | * d884ff9 Kamil Cudnik 250221:0109 - [syncd] Move logSet logGet under mutex to prevent race condition (sonic-net#1520) (sonic-net#1538) | * ec8b3c3 mssonicbld 250219:1902 - Fix pipeline errors related to rsyslogd and libswsscommon installation (sonic-net#1535) | * 6b263b8 mssonicbld 250219:0702 - [FC] Support Policer Counter (sonic-net#1533) | * e53489e Jianyue Wu 250219:0217 - [syncd] Update log level for bulk api (sonic-net#1532) | * 7ae00e5 mssonicbld 250211:0703 - Define bulk chunk size and bulk chunk size per counter ID (sonic-net#1528) | * f35e743 mssonicbld 250207:1005 - [nvidia] Skip SAI discovery on ports (sonic-net#1524) | * bf049ed mssonicbld 250205:1328 - Use sonictest pool instead of sonic-common and fix arm64 issue. (sonic-net#1516) | * ffe371d mssonicbld 250205:1327 - [syncd] Support bulk set in INIT_VIEW mode (sonic-net#1517) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I did
Do not enter the vendor SAI critical section for counter-polling/-clearing operations.
Why I did it
There are two types of threads in syncd:
The critical section in vendor SAI was introduced to protect vendors' SAI from being re-entered, which prevents the flex counter threads from running concurrently, and introduces latency for one flex counter thread when it's waiting for the critical section.
It is not necessary to enter a section in counter-polling/clearing operations since the objects' state won't be changed in that API.
How I verified it
Run regression test and observe counter-polling performance.
Details if related