Skip to content

Add orchagent heart beat message for watchdog. #2737

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jun 6, 2023
Merged

Add orchagent heart beat message for watchdog. #2737

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7b973dc
Add orchagent heartbeat message
liuh-80 Apr 17, 2023
f2be481
Change to use supervisord message
liuh-80 Apr 17, 2023
dd87f93
Update event message
liuh-80 Apr 17, 2023
392de22
Merge remote-tracking branch 'origin' into dev/liuh/add-heart-beat
liuh-80 Apr 23, 2023
ac63cf6
Improve heartbeat message
liuh-80 Apr 28, 2023
0c928c8
Improve code
liuh-80 Apr 28, 2023
b24ecb3
Imporve code by PR comments
liuh-80 May 15, 2023
12a3524
Improve code by PR comments
liuh-80 May 19, 2023
7f17fd4
Fix code
liuh-80 May 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions orchagent/orchdaemon.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
#include "logger.h"
#include <sairedis.h>
#include "warm_restart.h"
#include <iostream>

#define SAI_SWITCH_ATTR_CUSTOM_RANGE_BASE SAI_SWITCH_ATTR_CUSTOM_RANGE_START
#include "sairedis.h"
Expand All @@ -18,6 +19,9 @@ using namespace swss;
#define SELECT_TIMEOUT 1000
#define PFC_WD_POLL_MSECS 100

/* orchagent heart beat message interval */
#define HEART_BEAT_INTERVAL_MSECS 10 * 1000

extern sai_switch_api_t* sai_switch_api;
extern sai_object_id_t gSwitchId;
extern bool gSaiRedisLogRotate;
Expand Down Expand Up @@ -722,6 +726,7 @@ void OrchDaemon::start()
ret = m_select->select(&s, SELECT_TIMEOUT);

auto tend = std::chrono::high_resolution_clock::now();
heartBeat(tend);

auto diff = std::chrono::duration_cast<std::chrono::milliseconds>(tend - tstart);

Expand Down Expand Up @@ -958,6 +963,20 @@ void OrchDaemon::addOrchList(Orch *o)
m_orchList.push_back(o);
}

void OrchDaemon::heartBeat(std::chrono::time_point<std::chrono::high_resolution_clock> tcurrent)
{
static auto tlast = std::chrono::high_resolution_clock::now();
Copy link
Contributor

@qiluo-msft qiluo-msft May 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static

You are assuming OrchDaemon has only single instance in the process. To be super safe, you can use a static member variable instead of a static function variable. #Closed

Copy link
Contributor Author

@liuh-80 liuh-80 May 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, change to a static member variable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for misleading.

To be super safe, you can use a static member variable instead of a static function variable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, change to none static member.


// output heart beat message to SYSLOG
auto diff = std::chrono::duration_cast<std::chrono::milliseconds>(tcurrent - tlast);
if (diff.count() >= HEART_BEAT_INTERVAL_MSECS)
{
tlast = tcurrent;
// output heart beat message to supervisord with 'PROCESS_COMMUNICATION_STDOUT' event: http://supervisord.org/events.html
cout << "<!--XSUPERVISOR:BEGIN-->heartbeat<!--XSUPERVISOR:END-->" << endl;
}
}

FabricOrchDaemon::FabricOrchDaemon(DBConnector *applDb, DBConnector *configDb, DBConnector *stateDb, DBConnector *chassisAppDb) :
OrchDaemon(applDb, configDb, stateDb, chassisAppDb),
m_applDb(applDb),
Expand Down
2 changes: 2 additions & 0 deletions orchagent/orchdaemon.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,8 @@ class OrchDaemon
Select *m_select;

void flush();

void heartBeat(std::chrono::time_point<std::chrono::high_resolution_clock> tcurrent);
};

class FabricOrchDaemon : public OrchDaemon
Expand Down