Add syslog forwarding rules validation and thread exception handling #17572

ytzur1 · 2025-03-18T13:12:18Z

Add validation of rsyslog message from conf file like telemetry, bgp, gnmi, etc.
Implement thread exception handler for sub thread error capture

Description of PR

Adding validation of rsyslog messages from configuration files (telemetry, bgp, gnmi, etc.)
Implementing a thread exception handler for capturing subthread errors

Summary:
Fixes # (issue)

Type of change

Back port request

Approach

What is the motivation for this PR?

The current syslog testing framework lacks validation of rsyslog messages from various configuration files. Additionally, exception during subthreads were not being caught by the main thread.

How did you do it?

Added rsyslog message validation:
- Implemented verification of logger messages from telemetry, bgp, gnmi,
Implemented thread-safe exception handling:
- Created a global thread_exceptions list for capturing errors
- Added ThreadWrapper class for safe exception capture
- Implemented handle_thread_exceptions() for error reporting in main thread

How did you verify/test it?

Ran all test cases in test_syslog_source_ip.py

Any platform specific information?

No

Supported testbed topology if it's a new test case?

No

Documentation

Add validation of rsyslog message from conf file like telemetry, bgp, gnmi, etc. Implement thread-safe exception handler for subthread error capture Enhance logging message generation for comprehensive testing Tested: Ran all test cases in test_syslog_source_ip.py

mssonicbld · 2025-03-18T13:12:27Z

/azp run

azure-pipelines · 2025-03-18T13:12:38Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-18T17:01:39Z

/azp run

azure-pipelines · 2025-03-18T17:01:51Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-18T17:04:11Z

/azp run

azure-pipelines · 2025-03-18T17:04:23Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-18T17:10:57Z

/azp run

azure-pipelines · 2025-03-18T17:11:08Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-19T07:29:18Z

/azp run

azure-pipelines · 2025-03-19T07:29:30Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-19T08:37:57Z

/azp run

azure-pipelines · 2025-03-19T08:38:09Z

Azure Pipelines successfully started running 1 pipeline(s).

bingwang-ms · 2025-03-19T23:51:43Z

Related image change sonic-net/sonic-buildimage#21923

liuh-80 · 2025-03-20T01:47:18Z

tests/syslog/syslog_utils.py

-        time.sleep(0.2)
+    for flag, msg in logging_data:
+        for i in range(syslogUtilsConst.PACKETS_NUM):
+            dut.shell(f"logger {default_priority} {flag} {msg} {i + 1}")


This command run in host.

However, the config causes this issue are:
'''
if re_match($programname, "bgp[0-9]*#(frr|zebra|staticd|watchfrr)") then {
/var/log/frr/zebra.log
stop
}
'''

Which means when reproduce the issue, the log send inside bgp docker container, then the 'programname' part can match.

I warry about this test case can't prevent regression. can you test with an image have this issue to make sure this test can catch issue?

This is good suggestion: test with an image have this issue to make sure this test can catch issue

Tested with an image that had the configuration issue, and the test failed and reporting that not all types of forwarded messages were successful.

tests/syslog/test_syslog_source_ip.py

zbud-msft · 2025-03-25T23:28:27Z

Hi @ytzur1, I manually ran your test changes from our test server and seeing following issues when running the testcase. Can you please double check your logic in capture_syslog_packets() function in syslog/syslog_utils.py

25/03/2025 23:13:49 base._run                                L0108 DEBUG | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_Vrf-data.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_Vrf-data.pcap"}}, "_ansible_no_log": false, "changed": false}
25/03/2025 23:13:49 base._run                                L0108 DEBUG | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_default.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_default.pcap"}}, "_ansible_no_log": false, "changed": false}
25/03/2025 23:13:49 base._run                                L0108 DEBUG | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_mgmt.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_mgmt.pcap"}}, "_ansible_no_log": false, "changed": false}

zbud-msft

Check logic for capture_syslog_packets, seems to run into "file not found: /tmp/X.pcap" issue

mssonicbld · 2025-03-26T07:51:41Z

/azp run

azure-pipelines · 2025-03-26T07:51:52Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-26T07:53:58Z

/azp run

azure-pipelines · 2025-03-26T07:54:10Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-26T09:26:22Z

/azp run

azure-pipelines · 2025-03-26T09:26:34Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-26T10:22:20Z

/azp run

azure-pipelines · 2025-03-26T10:22:32Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-03-26T10:58:16Z

/azp run

azure-pipelines · 2025-03-26T10:58:28Z

Azure Pipelines successfully started running 1 pipeline(s).

ytzur1 · 2025-03-26T11:31:32Z

Hi, @zbud-msft
The error was caused by a missing tab in tcpdump cmd command after pre-commit formatting of long lines.
It's fixed and latest version was tested and it's working.

Hi @ytzur1, I manually ran your test changes from our test server and seeing following issues when running the testcase. Can you please double check your logic in capture_syslog_packets() function in syslog/syslog_utils.py

25/03/2025 23:13:49 base._run L0108 DEBUG | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_Vrf-data.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_Vrf-data.pcap"}}, "_ansible_no_log": false, "changed": false} 25/03/2025 23:13:49 base._run L0108 DEBUG | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_default.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_default.pcap"}}, "_ansible_no_log": false, "changed": false} 25/03/2025 23:13:49 base._run L0108 DEBUG | /var/src/sonic-mgmt-int/tests/syslog/syslog_utils.py::capture_syslog_packets#157: AnsibleModule::fetch Result => {"failed": true, "msg": "file not found: /tmp/test_syslog_tcpdump_mgmt.pcap", "invocation": {"module_args": {"src": "/tmp/test_syslog_tcpdump_mgmt.pcap"}}, "_ansible_no_log": false, "changed": false}

zbud-msft · 2025-03-28T00:42:59Z

@ytzur1 Would it be possible to post screenshot of testcases passing since kvm tests will skip?

qiluo-msft · 2025-03-28T23:32:43Z

tests/syslog/syslog_utils.py

+    default_priority = '--priority CRIT'
+    for flag, msg in logging_data:
+        for i in range(syslogUtilsConst.PACKETS_NUM):
+            dut.shell(f"logger {default_priority} {flag} {msg} {i + 1}")


logger

Could you also add testcase for this repro inside container:

admin@sonic:~$ docker exec -it bgp bash root@sonic:/# logger -p local0.notice -t bgpd TESTMESSAGE20

qiluo-msft · 2025-03-28T23:36:14Z

import pytest

Could ensure it is run in KVM test? so it will protect each PR.

Refers to: tests/syslog/test_syslog_source_ip.py:2 in e1ed921. [](commit_id = e1ed921, deletion_comment = False)

mssonicbld · 2025-03-31T15:04:21Z

/azp run

azure-pipelines · 2025-03-31T15:04:35Z

Azure Pipelines successfully started running 1 pipeline(s).

qiluo-msft

The active comments are non-blocking, susgest follow up with new PR. Merge to help enhance qualification pipeline to qualify recent image fix.

ytzur1 · 2025-04-01T06:21:10Z

@qiluo-msft

It seems the test_syslog_source_ip is included in kvm test but its being skipped

ytzur1 · 2025-04-01T06:58:11Z

@ytzur1 Would it be possible to post screenshot of testcases passing since kvm tests will skip?

…onic-net#17572) Add validation of rsyslog message from conf file like telemetry, bgp, gnmi, etc. Implement thread exception handler for sub thread error capture Description of PR Adding validation of rsyslog messages from configuration files (telemetry, bgp, gnmi, etc.) Implementing a thread exception handler for capturing subthread errors Approach What is the motivation for this PR? The current syslog testing framework lacks validation of rsyslog messages from various configuration files. Additionally, exception during subthreads were not being caught by the main thread. How did you do it? Added rsyslog message validation: Implemented verification of logger messages from telemetry, bgp, gnmi, Implemented thread-safe exception handling: Created a global thread_exceptions list for capturing errors Added ThreadWrapper class for safe exception capture Implemented handle_thread_exceptions() for error reporting in main thread How did you verify/test it? Ran all test cases in test_syslog_source_ip.py

roy-sror approved these changes Mar 18, 2025

View reviewed changes

Update syslog_utils.py

a854673

Update syslog_utils.py

6babf08

Update test_syslog_source_ip.py

5097f0c

qiluo-msft requested review from liuh-80 and zbud-msft March 18, 2025 19:25

Update test_syslog_source_ip.py

ec5e1e8

ytzur1 force-pushed the syslog_test_case branch from 537deea to ec5e1e8 Compare March 19, 2025 08:37

liuh-80 reviewed Mar 20, 2025

View reviewed changes

qiluo-msft reviewed Mar 21, 2025

View reviewed changes

tests/syslog/test_syslog_source_ip.py Show resolved Hide resolved

zbud-msft suggested changes Mar 25, 2025

View reviewed changes

fixing regex for rsyslog msgs

96de3e2

adding bgp#bgpd to forward msgs

57975c7

adding tab to tcpdump cmd

21c392a

verify tcpdump file was created

e1ed921

ytzur1 force-pushed the syslog_test_case branch from 501acf4 to e1ed921 Compare March 26, 2025 10:58

qiluo-msft reviewed Mar 28, 2025

View reviewed changes

Merge branch 'sonic-net:master' into syslog_test_case

cd7790f

qiluo-msft approved these changes Mar 31, 2025

View reviewed changes

zbud-msft approved these changes Mar 31, 2025

View reviewed changes

qiluo-msft merged commit ba794a1 into sonic-net:master Mar 31, 2025
13 checks passed

Add syslog forwarding rules validation and thread exception handling #17572

Add syslog forwarding rules validation and thread exception handling #17572

Conversation

ytzur1 commented Mar 18, 2025 • edited Loading

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld commented Mar 18, 2025

azure-pipelines bot commented Mar 18, 2025

mssonicbld commented Mar 18, 2025

azure-pipelines bot commented Mar 18, 2025

mssonicbld commented Mar 18, 2025

azure-pipelines bot commented Mar 18, 2025

mssonicbld commented Mar 18, 2025

azure-pipelines bot commented Mar 18, 2025

mssonicbld commented Mar 19, 2025

azure-pipelines bot commented Mar 19, 2025

mssonicbld commented Mar 19, 2025

azure-pipelines bot commented Mar 19, 2025

bingwang-ms commented Mar 19, 2025

liuh-80 Mar 20, 2025

Choose a reason for hiding this comment

qiluo-msft Mar 21, 2025

Choose a reason for hiding this comment

ytzur1 Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

zbud-msft commented Mar 25, 2025

zbud-msft left a comment

Choose a reason for hiding this comment

mssonicbld commented Mar 26, 2025

azure-pipelines bot commented Mar 26, 2025

mssonicbld commented Mar 26, 2025

azure-pipelines bot commented Mar 26, 2025

mssonicbld commented Mar 26, 2025

azure-pipelines bot commented Mar 26, 2025

mssonicbld commented Mar 26, 2025

azure-pipelines bot commented Mar 26, 2025

mssonicbld commented Mar 26, 2025

azure-pipelines bot commented Mar 26, 2025

ytzur1 commented Mar 26, 2025 • edited Loading

zbud-msft commented Mar 28, 2025

qiluo-msft Mar 28, 2025 • edited Loading

Choose a reason for hiding this comment

qiluo-msft commented Mar 28, 2025

mssonicbld commented Mar 31, 2025

azure-pipelines bot commented Mar 31, 2025

qiluo-msft left a comment

Choose a reason for hiding this comment

ytzur1 commented Apr 1, 2025 • edited Loading

ytzur1 commented Apr 1, 2025 • edited Loading

ytzur1 commented Mar 18, 2025 •

edited

Loading

ytzur1 Mar 26, 2025 •

edited

Loading

ytzur1 commented Mar 26, 2025 •

edited

Loading

qiluo-msft Mar 28, 2025 •

edited

Loading

ytzur1 commented Apr 1, 2025 •

edited

Loading

ytzur1 commented Apr 1, 2025 •

edited

Loading