[Fastboot] Delay PMON service for better fastboot performance #10567

shlomibitton · 2022-04-13T15:35:00Z

Signed-off-by: Shlomi Bitton [email protected]

Why I did it

Profiling the system state on init after fast-reboot during create_switch function execution, it is possible to see few python scripts running at the same time.
This parallel execution consume CPU time and the duration of create_switch is longer than it should be.
Following this finding, and the motivation to ensure these services will not interfere in the future, PMON is delayed in 90 seconds until the system finish the init flow after fastboot.

How I did it

Add a timer for PMON service.
Exclude for MLNX platform the start trigger of PMON when SYNCD starts in case of fastboot.
Copy the timer file to the host bin image.

How to verify it

Run fast-reboot on MLNX platform and observe faster create_switch execution time.

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Shlomi Bitton <[email protected]>

stepanblyschak · 2022-04-14T13:35:14Z

files/scripts/syncd.sh

+        BOOT_TYPE=`getBootType`
+        if [[ x"$BOOT_TYPE" == x"fast" ]]; then
+            return
+        fi
        debug "Starting pmon service..."
        /bin/systemctl start pmon
        debug "Started pmon service"


This was added by #3505 to fix some issues related to pmon/SDK. Won't that issue happen again with your change?

No, notice my change is only on startup flow and not shutdown flow.
Delaying PMON is OK as long SYNCD is running before starting it.
For shutdown flow the behavior is the same.

This change can be made generic to all platforms. CC @yxieca.

Also, if this is not made generic, then the comments on files/build_templates/pmon.timer should be modified to say:
# This delay is for fast-reboot performance on MLNX platform

Also, don't you want to do this change for warm-reboot as well? I am curious why only for fast-reboot?

It is a generic change, this exception is only for MLNX platform, so I added a condition for it in order to keep the delay and not start PMON with SYNCD start flow.
Regarding warm-boot, I totally agree we can do it for both types, fixed it.

vaibhavhd · 2022-04-14T16:55:58Z

files/scripts/syncd.sh

@@ -46,6 +46,10 @@ function startplatform() {
 function waitplatform() {

    if [[ x"$sonic_asic_platform" == x"mellanox" ]]; then
+        BOOT_TYPE=`getBootType`
+        if [[ x"$BOOT_TYPE" == x"fast" ]]; then
+            return


It would be meaningful to have a log line here mentioning that for fastboot (or warm), pmon start will be delayed as governed by files/build_templates/pmon.timer

vaibhavhd · 2022-04-18T20:20:03Z

files/build_templates/pmon.timer

@@ -0,0 +1,12 @@
+[Unit]
+# This delay is for fast-reboot performance


Correction - fast-reboot and warm-reboot performance.

vaibhavhd · 2022-04-18T21:00:43Z

files/scripts/syncd.sh

-        debug "Starting pmon service..."
-        /bin/systemctl start pmon
-        debug "Started pmon service"
+        if [[ x"$BOOT_TYPE" != x"cold" ]]; then


Same comment as on lldp PR:
Minor: This is correct, but it may be better to match against whitelist BOOT_TYPEs (fast, warm, fastfast).
This is so that this condition does not incorrectly evaluate to true for any other boottype that gets added in future.

vaibhavhd

Minor comments. LGTM otherwise.

Fix review comments.

shlomibitton · 2022-04-20T10:23:57Z

@vaibhavhd @stepanblyschak I fixed your comments, can you please review and approve?
Thanks

liat-grozovik · 2022-04-22T08:04:40Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2022-04-22T08:04:53Z

Azure Pipelines successfully started running 1 pipeline(s).

shlomibitton · 2022-04-25T08:02:58Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-04-25T08:02:59Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-04-25T08:03:11Z

Azure Pipelines successfully started running 1 pipeline(s).

liat-grozovik · 2022-04-25T09:12:07Z

@vaibhavhd kindly reminder to review

liat-grozovik · 2022-04-28T07:30:55Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2022-04-28T07:31:08Z

Azure Pipelines successfully started running 1 pipeline(s).

- Why I did it Profiling the system state on init after fast-reboot during create_switch function execution, it is possible to see few python scripts running at the same time. This parallel execution consume CPU time and the duration of create_switch is longer than it should be. Following this finding, and the motivation to ensure these services will not interfere in the future, LLDP is delayed in 90 seconds until the system finish the init flow after fastboot. - How I did it Add a timer for LLDP service. Copy the timer file to the host bin image. - How to verify it Run fast-reboot on MLNX platform and observe faster create_switch execution time. This PR is dependent on PR: #10567

liat-grozovik · 2022-05-02T07:44:49Z

As prev approved by @vaibhavhd and merge only done to avoid checkers failures, PR is now merged.

qiluo-msft · 2022-05-03T00:43:40Z

This commit could not be cleanly cherry-picked to 202012. Please submit another PR.

shlomibitton · 2022-05-04T08:07:10Z

@qiluo-msft I created a separate PR for 202012:
#10745

- Why I did it Profiling the system state on init after fast-reboot during create_switch function execution, it is possible to see few python scripts running at the same time. This parallel execution consume CPU time and the duration of create_switch is longer than it should be. Following this finding, and the motivation to ensure these services will not interfere in the future, PMON is delayed in 90 seconds until the system finish the init flow after fastboot. - How I did it Add a timer for PMON service. Exclude for MLNX platform the start trigger of PMON when SYNCD starts in case of fastboot. Copy the timer file to the host bin image. - How to verify it Run fast-reboot on MLNX platform and observe faster create_switch execution time.

…net#10568) - Why I did it Profiling the system state on init after fast-reboot during create_switch function execution, it is possible to see few python scripts running at the same time. This parallel execution consume CPU time and the duration of create_switch is longer than it should be. Following this finding, and the motivation to ensure these services will not interfere in the future, LLDP is delayed in 90 seconds until the system finish the init flow after fastboot. - How I did it Add a timer for LLDP service. Copy the timer file to the host bin image. - How to verify it Run fast-reboot on MLNX platform and observe faster create_switch execution time. This PR is dependent on PR: sonic-net#10567

…net#10567) - Why I did it Profiling the system state on init after fast-reboot during create_switch function execution, it is possible to see few python scripts running at the same time. This parallel execution consume CPU time and the duration of create_switch is longer than it should be. Following this finding, and the motivation to ensure these services will not interfere in the future, PMON is delayed in 90 seconds until the system finish the init flow after fastboot. - How I did it Add a timer for PMON service. Exclude for MLNX platform the start trigger of PMON when SYNCD starts in case of fastboot. Copy the timer file to the host bin image. - How to verify it Run fast-reboot on MLNX platform and observe faster create_switch execution time.

Delay PMON service for better fastboot performance

18c1516

Signed-off-by: Shlomi Bitton <[email protected]>

shlomibitton requested a review from lguohan as a code owner April 13, 2022 15:35

shlomibitton mentioned this pull request Apr 13, 2022

[fast-reboot] Optimize PMON and LLDP services start up CPU consumption #10242

Closed

6 tasks

liat-grozovik added the Request for 202111 Branch For PRs being requested for 202111 branch label Apr 13, 2022

liat-grozovik requested review from stepanblyschak and vaibhavhd and removed request for lguohan April 13, 2022 15:47

yxieca requested review from vdahiya12 and zjswhhh April 13, 2022 20:40

stepanblyschak reviewed Apr 14, 2022

View reviewed changes

vaibhavhd reviewed Apr 14, 2022

View reviewed changes

Shlomi Bitton added 2 commits April 17, 2022 08:33

Delay PMON in warm-boot case also

bcefed7

Add a log line indicating the delay reason

7e0c3cd

shlomibitton requested review from stepanblyschak and vaibhavhd April 17, 2022 08:41

Delay PMON for any boot type except cold boot

1c00f14

shlomibitton mentioned this pull request Apr 17, 2022

[Fastboot] Delay LLDP service for better fastboot performance #10568

Merged

6 tasks

liat-grozovik added the Request for 202012 Branch label Apr 18, 2022

vaibhavhd reviewed Apr 18, 2022

View reviewed changes

vaibhavhd previously approved these changes Apr 18, 2022

View reviewed changes

Allign init_cfg.json.j2 to delay PMON.

0f332e3

Fix review comments.

shlomibitton dismissed vaibhavhd’s stale review via 0f332e3 April 20, 2022 10:21

shlomibitton requested a review from vaibhavhd April 20, 2022 10:22

stepanblyschak previously approved these changes Apr 22, 2022

View reviewed changes

vaibhavhd previously approved these changes Apr 28, 2022

View reviewed changes

Merge branch 'master' into shlomi_delay_pmon_service

0b84c45

shlomibitton dismissed stale reviews from vaibhavhd and stepanblyschak via 0b84c45 May 1, 2022 10:57

shlomibitton requested review from stepanblyschak and vaibhavhd May 1, 2022 10:57

liat-grozovik approved these changes May 2, 2022

View reviewed changes

liat-grozovik merged commit 4ec3af8 into sonic-net:master May 2, 2022

shlomibitton mentioned this pull request May 4, 2022

[202012] [Fastboot] Delay LLDP service for better fastboot performance #10744

Merged

6 tasks

judyjoseph added the Included in 202111 Branch label May 8, 2022

qiluo-msft removed the Request for 202012 Branch label May 10, 2022

shlomibitton deleted the shlomi_delay_pmon_service branch May 26, 2022 12:25

arfeigin mentioned this pull request Jul 12, 2023

Fix issue: set has_timer attribute to true for platform monitor service #15711

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fastboot] Delay PMON service for better fastboot performance #10567

[Fastboot] Delay PMON service for better fastboot performance #10567

shlomibitton commented Apr 13, 2022 •

edited

Loading

stepanblyschak Apr 14, 2022

shlomibitton Apr 14, 2022 •

edited

Loading

vaibhavhd Apr 14, 2022

vaibhavhd Apr 14, 2022

shlomibitton Apr 17, 2022

vaibhavhd Apr 14, 2022

shlomibitton Apr 17, 2022

vaibhavhd Apr 18, 2022

vaibhavhd Apr 18, 2022

vaibhavhd left a comment

shlomibitton commented Apr 20, 2022

liat-grozovik commented Apr 22, 2022

azure-pipelines bot commented Apr 22, 2022

shlomibitton commented Apr 25, 2022

mssonicbld commented Apr 25, 2022

azure-pipelines bot commented Apr 25, 2022

liat-grozovik commented Apr 25, 2022

liat-grozovik commented Apr 28, 2022

azure-pipelines bot commented Apr 28, 2022

liat-grozovik commented May 2, 2022

qiluo-msft commented May 3, 2022

shlomibitton commented May 4, 2022

		@@ -0,0 +1,12 @@
		[Unit]
		# This delay is for fast-reboot performance

[Fastboot] Delay PMON service for better fastboot performance #10567

[Fastboot] Delay PMON service for better fastboot performance #10567

Conversation

shlomibitton commented Apr 13, 2022 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Choose a reason for hiding this comment

shlomibitton Apr 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vaibhavhd left a comment

Choose a reason for hiding this comment

shlomibitton commented Apr 20, 2022

liat-grozovik commented Apr 22, 2022

azure-pipelines bot commented Apr 22, 2022

shlomibitton commented Apr 25, 2022

mssonicbld commented Apr 25, 2022

azure-pipelines bot commented Apr 25, 2022

liat-grozovik commented Apr 25, 2022

liat-grozovik commented Apr 28, 2022

azure-pipelines bot commented Apr 28, 2022

liat-grozovik commented May 2, 2022

qiluo-msft commented May 3, 2022

shlomibitton commented May 4, 2022

shlomibitton commented Apr 13, 2022 •

edited

Loading

shlomibitton Apr 14, 2022 •

edited

Loading