Skip to content

Revert "Add watchdog mechanism to swss service and generate alert when swss have issue." #15390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

yejianquan
Copy link
Contributor

@yejianquan yejianquan commented Jun 8, 2023

Reverts #14686

process_monitoring/test_critical_process_monitoring.py failed because there is no file ‘/etc/supervisor/critical_processes’ in the bgp and other containers, so the ‘supervisor-proc-exit-listener’ can’t up:

In the bgp container:

root@vlab-01:/# supervisorctl status
bgpcfgd RUNNING pid 52, uptime 0:12:13
bgpd RUNNING pid 50, uptime 0:12:13
bgpmon RUNNING pid 58, uptime 0:12:13
containercfgd RUNNING pid 36, uptime 0:12:14
dependent-startup EXITED Jun 08 10:22 AM
fpmsyncd RUNNING pid 59, uptime 0:12:13
rsyslogd RUNNING pid 31, uptime 0:12:15
staticd RUNNING pid 49, uptime 0:12:13
staticroutebfd RUNNING pid 60, uptime 0:12:13
supervisor-proc-exit-listener FATAL Exited too quickly (process log may have details)
zebra RUNNING pid 35, uptime 0:12:14
zsocket EXITED Jun 08 10:22 AM
root@vlab-01:/#

root@vlab-01:/# /usr/bin/supervisor-proc-exit-listener --container-name bgp
Traceback (most recent call last):
File "/usr/bin/supervisor-proc-exit-listener", line 212, in
main(sys.argv[1:])
File "/usr/bin/supervisor-proc-exit-listener", line 135, in main
_, watch_process_list = get_group_and_process_list(WATCH_PROCESSES_FILE)
File "/usr/bin/supervisor-proc-exit-listener", line 51, in get_group_and_process_list
with open(process_file, 'r') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/supervisor/watchdog_processes'
root@vlab-01:/#

The watchdog_processes is introduced by: Add watchdog mechanism to swss service and generate alert when swss have issue. by liuh-80 · Pull Request #14686 · sonic-net/sonic-buildimage (github.com)

But a weird thing is, the test still failed on my try-revert pr: Revert "Add watchdog mechanism to swss service and generate alert when swss have issue." by yejianquan · Pull Request #15390 · sonic-net/sonic-buildimage (github.com)
I debugged in the image built by the revert PR, the ‘supervisor-proc-exit-listener’ is still old that wants to load ‘/etc/supervisor/watchdog_processes'.

I’m pretty sure the failure is caused the watchdog_processes, if I manually comment related code in the kvm testbed, the ‘supervisor-proc-exit-listener’ can up as expected.

@yejianquan yejianquan requested a review from lguohan as a code owner June 8, 2023 07:27
@yejianquan yejianquan marked this pull request as draft June 8, 2023 07:28
@yejianquan yejianquan marked this pull request as ready for review June 9, 2023 00:28
@liuh-80 liuh-80 self-requested a review June 9, 2023 00:31
@liuh-80
Copy link
Contributor

liuh-80 commented Jun 9, 2023

The issue seems caused by PR merge validation, the build image will use old docker image so the PR merge validation not found origial PR issue and this PR still failed.

@liuh-80 liuh-80 requested a review from qiluo-msft June 9, 2023 00:50
@wangxin wangxin merged commit cec9d7b into sonic-net:master Jun 9, 2023
sonic-otn pushed a commit to sonic-otn/sonic-buildimage that referenced this pull request Sep 20, 2023
…n swss have issue. (sonic-net#14686)" (sonic-net#15390)

This reverts commit 44427a2.
Docker image not updated during PR validation and caused PR check failures.
Force merge this revert. After cache is updated after this PR is merged, issue should be fixed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants