[Accton] Support system health monitoring on pddf implemented models #8198

seanwu-ec · 2021-07-16T06:48:22Z

Why I did it

To support SystemHealthMonitoring feature on Accton's models (which have implemented PDDF APIs).

How I did it

Provided a config file, system_health_monitoring_config.json.
Implement the SystemLED part that'd be manipulated by HealthChecker.
Enable thermalctld that makes more HW checks available

How to verify it

Use show system-health detail and check if all desired checks pass, and confirm if systemLED works reasonably.

////// SEAN: Booting, LED_DIAG = GREEN_BLINKING //////

root@sonic:/home/admin# show system-health detail 
System is currently booting...


////// SEAN: verifying with PSU2 unplugged //////
////// SEAN: Failed the health check, LED_DIAG = AMBER //////

root@sonic:/home/admin# show system-health detail
System status summary

  System status LED  amber
  Services:
    Status: OK
  Hardware:
    Status: Not OK
    Reasons: PSU 2 is missing or not available
	     PSU2_FAN1 is broken

System services and devices monitor list

Name                        Status    Type
--------------------------  --------  ----------
sonic                       OK        System
rsyslog                     OK        Process
root-overlay                OK        Filesystem
var-log                     OK        Filesystem
routeCheck                  OK        Program
diskCheck                   OK        Program
container_checker           OK        Program
container_memory_telemetry  OK        Program
PSU2_FAN1                   Not OK    Fan
PSU 2                       Not OK    PSU
PSU1_FAN1                   OK        Fan
PSU 1                       OK        PSU

System services and devices ignore list

Name             Status    Type
---------------  --------  ------
PSU1_FAN1.speed  Ignored   Device
PSU2_FAN1.speed  Ignored   Device
asic             Ignored   Device
psu.voltage      Ignored   Device
psu.temperature  Ignored   Device


////// SEAN: Ignore PSU2 in the config file. //////
////// SEAN: Passed the health check, LED_DIAG = GREEN //////

root@sonic:/home/admin# vi /usr/share/sonic/device/x86_64-accton_as7326_56x-r0/system_health_monitoring_config.json 
root@sonic:/home/admin# show system-health detail
System status summary

  System status LED  green
  Services:
    Status: OK
  Hardware:
    Status: OK

System services and devices monitor list

Name                        Status    Type
--------------------------  --------  ----------
sonic                       OK        System
rsyslog                     OK        Process
root-overlay                OK        Filesystem
var-log                     OK        Filesystem
routeCheck                  OK        Program
diskCheck                   OK        Program
container_checker           OK        Program
container_memory_telemetry  OK        Program
PSU1_FAN1                   OK        Fan
PSU 1                       OK        PSU

System services and devices ignore list

Name             Status    Type
---------------  --------  ------
psu.voltage      Ignored   Device
PSU1_FAN1.speed  Ignored   Device
PSU 2            Ignored   Device
asic             Ignored   Device
PSU2_FAN1        Ignored   Device
psu.temperature  Ignored   Device
PSU2_FAN1.speed  Ignored   Device

Which release branch to backport (provide reason below if selected)

Description for the changelog

[Accton] Support system health monitoring on pddf implemented models

Models: AS9716-32D, AS7326-56X, AS7726-32X, AS7816-64X

A picture of a cute animal (not mandatory but encouraged)

1. Implement SysLED on chassis. 2. Add config to system health monitor. Signed-off-by: Sean Wu <[email protected]>

1. Workaround to PddfFan.get_target_speed(), where always returns 0 for PSU's fan. NOTE: No fans get enumerated by thermalctld because PDDF hasn't implemented FanDrawer yet. Signed-off-by: Sean Wu <[email protected]>

Enable thermalctld for more HW checks. Signed-off-by: Sean Wu <[email protected]>

Enable thermalctld for more HW checks Signed-off-by: Sean Wu <[email protected]>

1. Enable thermalctld for more HW checks 2. Correct LED attr_name in pddf-device.json Signed-off-by: Sean Wu <[email protected]>

Models: AS7326-56X, AS7726-32X, AS9716-32D Signed-off-by: Sean Wu <[email protected]>

seanwu-ec · 2021-07-19T01:50:45Z

/azpw run

mssonicbld · 2021-07-19T01:50:46Z

/AzurePipelines run

azure-pipelines · 2021-07-19T01:51:05Z

Azure Pipelines successfully started running 1 pipeline(s).

seanwu-ec · 2021-08-03T08:07:40Z

/azpw run

mssonicbld · 2021-08-03T08:07:41Z

/AzurePipelines run

azure-pipelines · 2021-08-03T08:08:01Z

Azure Pipelines successfully started running 1 pipeline(s).

FuzailBrcm · 2021-08-20T05:29:56Z

platform/broadcom/sonic-platform-modules-accton/as9716-32d/sonic_platform/fan.py

@@ -46,3 +46,9 @@ def get_direction(self):

        return direction

+    def get_target_speed(self):
+        if self.is_psu_fan:
+            raise NotImplementedError # Target speed not supported for PSU fans


@seanwu-ec
Do you think its better to implement this is the common pddf_fan.py ?

# git diff platform/pddf/platform-api-pddf-base/sonic_platform_pddf_base/pddf_fan.py diff --git a/platform/pddf/platform-api-pddf-base/sonic_platform_pddf_base/pddf_fan.py b/platform/pddf/platform-api-pddf-base/sonic_platform_pddf_base/pddf_fan.py index 30be0dd74..d272d40e6 100755 --- a/platform/pddf/platform-api-pddf-base/sonic_platform_pddf_base/pddf_fan.py +++ b/platform/pddf/platform-api-pddf-base/sonic_platform_pddf_base/pddf_fan.py @@ -224,7 +224,7 @@ class PddfFan(FanBase): target_speed = 0 if self.is_psu_fan: # Target speed not usually supported for PSU fans - target_speed = 0 + raise NotImplementedError else: idx = (self.fantray_index-1)*self.platform['num_fans_pertray'] + self.fan_index attr = "fan" + str(idx) + "_pwm" #

Yes, because PSU doesn't reveal target fan speed. So I think it's better pddf_fan.py implement this.
I've raised an issue for that sometime before: #8129

In order to pass the fan status check at https://github.com/Azure/sonic-platform-daemons/blob/master/sonic-thermalctld/scripts/thermalctld#L341, according to the try_get() in thermalctld, there are two ways of doing this. One is raise NotImplementedError another one is return None. You can go whichever you like.

Thanks @seanwu-ec
I added the fix in common PDDF APIs as part of PR (#7834)

You can remove those changes from individual platform's fan.py

Done. 😊 Thanks, @FuzailBrcm

* Have moved that implementation to PR#7834 Signed-off-by: Sean Wu <[email protected]>

seanwu-ec · 2022-07-08T08:00:44Z

some codes were merged by PR#10053, PR#9982 and PR#8164. so close this PR.

seanwu-ec added 6 commits July 12, 2021 06:35

[AS9716-32D] Support system health monitoring

823e8c5

1. Implement SysLED on chassis. 2. Add config to system health monitor. Signed-off-by: Sean Wu <[email protected]>

[AS9716-32D] Enable thermalctld

7c0304c

1. Workaround to PddfFan.get_target_speed(), where always returns 0 for PSU's fan. NOTE: No fans get enumerated by thermalctld because PDDF hasn't implemented FanDrawer yet. Signed-off-by: Sean Wu <[email protected]>

[AS7326-56X] Support system health monitoring

a9a9869

Enable thermalctld for more HW checks. Signed-off-by: Sean Wu <[email protected]>

[AS7816-64X] Support system health monitoring

377c381

Enable thermalctld for more HW checks Signed-off-by: Sean Wu <[email protected]>

[as7726-32X] Support system health monitoring

86a1a83

1. Enable thermalctld for more HW checks 2. Correct LED attr_name in pddf-device.json Signed-off-by: Sean Wu <[email protected]>

Disable pcied on a few models

62b6165

Models: AS7326-56X, AS7726-32X, AS9716-32D Signed-off-by: Sean Wu <[email protected]>

seanwu-ec requested review from jleveque and lguohan as code owners July 16, 2021 06:48

Merge branch 'master' into pddf_models_support_system_health

0de9932

FuzailBrcm reviewed Aug 20, 2021

View reviewed changes

Rollback PSU fan workaround on get_target_speed

e52db48

* Have moved that implementation to PR#7834 Signed-off-by: Sean Wu <[email protected]>

FuzailBrcm approved these changes Aug 25, 2021

View reviewed changes

seanwu-ec requested a review from a team as a code owner June 10, 2022 02:01

seanwu-ec closed this Jul 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Accton] Support system health monitoring on pddf implemented models #8198

[Accton] Support system health monitoring on pddf implemented models #8198

Uh oh!

seanwu-ec commented Jul 16, 2021 •

edited

Loading

Uh oh!

seanwu-ec commented Jul 19, 2021

Uh oh!

mssonicbld commented Jul 19, 2021

Uh oh!

azure-pipelines bot commented Jul 19, 2021

Uh oh!

seanwu-ec commented Aug 3, 2021

Uh oh!

mssonicbld commented Aug 3, 2021

Uh oh!

azure-pipelines bot commented Aug 3, 2021

Uh oh!

FuzailBrcm Aug 20, 2021 •

edited

Loading

Uh oh!

seanwu-ec Aug 20, 2021

Uh oh!

FuzailBrcm Aug 23, 2021

Uh oh!

FuzailBrcm Aug 23, 2021

Uh oh!

seanwu-ec Aug 25, 2021

Uh oh!

seanwu-ec commented Jul 8, 2022

Uh oh!

Uh oh!

[Accton] Support system health monitoring on pddf implemented models #8198

[Accton] Support system health monitoring on pddf implemented models #8198

Uh oh!

Conversation

seanwu-ec commented Jul 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

seanwu-ec commented Jul 19, 2021

Uh oh!

mssonicbld commented Jul 19, 2021

Uh oh!

azure-pipelines bot commented Jul 19, 2021

Uh oh!

seanwu-ec commented Aug 3, 2021

Uh oh!

mssonicbld commented Aug 3, 2021

Uh oh!

azure-pipelines bot commented Aug 3, 2021

Uh oh!

FuzailBrcm Aug 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seanwu-ec Aug 20, 2021

Choose a reason for hiding this comment

Uh oh!

FuzailBrcm Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

FuzailBrcm Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

seanwu-ec Aug 25, 2021

Choose a reason for hiding this comment

Uh oh!

seanwu-ec commented Jul 8, 2022

Uh oh!

Uh oh!

seanwu-ec commented Jul 16, 2021 •

edited

Loading

FuzailBrcm Aug 20, 2021 •

edited

Loading