[FDB]All MACs are not synced to the kernel in scale scenario #12502

dgsudharsan · 2022-10-26T04:23:51Z

Description

During scale testing, all MACs are not synced to kernel. This can be easily reproduced by learning 10K MACs on a port and doing a shutdown/no shut. Could this be due to fdbsyncd not checking if port is up in kernel before programming the mac?

The second scenario where the problem happens is when MAC ages out in kernel but not in switch (if FDB aging time is increased in switch). This results in fdbsyncd processing stale mac notifications and reprogramming them based on status in state_db. In this scenario it is always observed that exactly 8K macs are reprogrammed (could be size of netlink buffer queue?)

The above two scenarios will result in many MACs not synced to remote VTEPS in EVPN.

Steps to reproduce the issue:

Learn 10 K MACs in a port
Shutdown the interface
Startup the interface.

Describe the results you received:

Describe the results you expected:

Output of `show version`:

SONiC Software Version: SONiC.202205.42-ea51d9514_Internal
Distribution: Debian 11.5
Kernel: 5.10.0-12-2-amd64
Build commit: ea51d9514
Build date: Fri Oct  7 05:45:56 UTC 2022
Built by: sw-r2d2-bot@r-build-sonic-ci03-243

Platform: x86_64-mlnx_msn2700-r0
HwSKU: Mellanox-SN2700
ASIC: mellanox
ASIC Count: 1
Serial Number: MT1829X20804
Model Number: MSN2700-CB2F
Hardware Revision: A2
Uptime: 07:07:08 up 11:30,  3 users,  load average: 2.17, 2.46, 2.58
Date: Wed 26 Oct 2022 07:07:08

Docker images:
REPOSITORY                                         TAG                            IMAGE ID       SIZE
docker-syncd-mlnx                                  202205.42-ea51d9514_Internal   97dd12cebe1e   859MB
docker-syncd-mlnx                                  latest                         97dd12cebe1e   859MB
docker-orchagent                                   202205.42-ea51d9514_Internal   347f0cdc723f   478MB
docker-orchagent                                   latest                         347f0cdc723f   478MB
docker-fpm-frr                                     202205.42-ea51d9514_Internal   ddadceae2d69   488MB
docker-fpm-frr                                     latest                         ddadceae2d69   488MB
docker-teamd                                       202205.42-ea51d9514_Internal   28f79f968d3c   459MB
docker-teamd                                       latest                         28f79f968d3c   459MB
docker-platform-monitor                            202205.42-ea51d9514_Internal   629c9ea03cf2   861MB
docker-platform-monitor                            latest                         629c9ea03cf2   861MB
docker-macsec                                      latest                         a7ea8b95281f   461MB
docker-snmp                                        202205.42-ea51d9514_Internal   0e96a62d07ee   488MB
docker-snmp                                        latest                         0e96a62d07ee   488MB
docker-dhcp-relay                                  latest                         8cef09a39edf   452MB
docker-lldp                                        202205.42-ea51d9514_Internal   337146c6b971   485MB
docker-lldp                                        latest                         337146c6b971   485MB
docker-mux                                         202205.42-ea51d9514_Internal   464339799d55   492MB
docker-mux                                         latest                         464339799d55   492MB
docker-sonic-telemetry                             202205.42-ea51d9514_Internal   7fc604d28c7c   523MB
docker-sonic-telemetry                             latest                         7fc604d28c7c   523MB
docker-database                                    202205.42-ea51d9514_Internal   98a7bdcfd7e8   443MB
docker-database                                    latest                         98a7bdcfd7e8   443MB
docker-router-advertiser                           202205.42-ea51d9514_Internal   f05c810acb38   443MB
docker-router-advertiser                           latest                         f05c810acb38   443MB
docker-nat                                         202205.42-ea51d9514_Internal   272fda2cdf1a   430MB
docker-nat                                         latest                         272fda2cdf1a   430MB
docker-sflow                                       202205.42-ea51d9514_Internal   5723c8d63918   428MB
docker-sflow                                       latest                         5723c8d63918   428MB
docker-sonic-mgmt-framework                        202205.42-ea51d9514_Internal   0fd3a3d91b98   557MB
docker-sonic-mgmt-framework                        latest                         0fd3a3d91b98   557MB
urm.nvidia.com/sw-nbu-sws-sonic-docker/sonic-wjh   1.3.1-202205                   4e8b9199b984   643MB

Output of `show techsupport`:

(paste your output here or download and attach the file here )

Issue 1 - sonic_dump_qa-eth-vt05-2-2700a1_20221026_070014
Issue 2 - sonic_dump_qa-eth-vt03-2-3700v_20221023_204313

Additional information you deem important (e.g. issue happens only occasionally):

sonic_dump_qa-eth-vt05-2-2700a1_20221026_070014.tar.gz
sonic_dump_qa-eth-vt03-2-3700v_20221023_204313.tar.gz

The text was updated successfully, but these errors were encountered:

adyeung · 2022-10-28T23:58:00Z

@kishorekunal01 from BRCM to take a look

kishorekunal01 · 2022-10-29T00:13:38Z

This is not issue, this is kernel behavior, and reprogramming is done so that MAC will be added back to kernel as per design.

The second scenario where the problem happens is when MAC ages out in kernel but not in switch (if FDB aging time is increased in switch). This results in fdbsyncd processing stale mac notifications and reprogramming them based on status in state_db. In this scenario it is always observed that exactly 8K macs are reprogrammed (could be size of netlink buffer queue?)

kishorekunal01 · 2022-10-29T00:35:18Z

@dgsudharsan

When interface goes down all the entry in the FDB_TABLE for the interface should get deleted by the switch, And when interface comes up(in kernel also interface will come up) and now due to traffic MAC learning will again happen in ASIC and same will be added back in FDB_TABLE.

If it is not working please enable below debug "swssloglevel -l INFO -c fdbsyncd" and provide the tech support.

dgsudharsan · 2022-10-31T21:11:12Z

@kishorekunal01 The issue is not with reprogramming the MACs but the consistency of mac between ASIC and kernel. One of the dumps captured has swssloglevel enabled as info. As mentioned there are two scenarios where macs are not properly reprogrammed in the kernel. I believe fdbsyncd is not robust enough to handle these scenarios. Please let me know if you need more details.

kishorekunal01 · 2022-10-31T23:24:01Z

@dgsudharsan
As per above reply, the issue here is local MAC is not in sync from ASIC to Kernel.

I tried the interface up/down test case on Broadcom chipset. And I don't see any issue with MAC sync between the ASIC and kernel.

Attaching Tech support in next comment

As I have earlier replied
Step 1: When interface goes down all the entry in the FDB_TABLE for the interface should get deleted(Trigger from ASIC)
Step2: When interface comes up with command "config interface startup Ethernetxx" kernel also interface will come up.
Step3: Due to traffic MAC learning will again happen in ASIC and same will be added back in FDB_TABLE.
Step4: Fdbsyncd will sync this MAC(FDB_TABLE) to the Kernel.

Testing_With10K_MAC_Broadcom_Chip.log

kishorekunal01 · 2022-10-31T23:35:54Z

I have enable debug "swssloglevel -l INFO -c fdbsyncd" and collected the tech support. Log file attached

sonic_dump_sonic_20221031_230704.zip

dgsudharsan · 2022-11-03T03:15:25Z

@kishorekunal01 Thanks. For the port up down scenario, I did some more analysis and found the rc to be due to SAI notification issue which I am handling internally.

However I also specified the netlink buffer issue which you can see from the logs

Oct 23 20:24:52.707880 qa-eth-vt03-2-3700v ERR swss#fdbsyncd: :- readData: netlink reports out of memory on reading a netlink socket. High possibility of a lost message

Please check sonic_dump_qa-eth-vt03-2-3700v_20221023_204313.tar attached in the bug. This happens when I have 10K macs on one port.

kishorekunal01 · 2022-11-15T01:21:18Z

@prsunny Can we increase the netlink buffer. Currently it is set to 3MB, Can we increase the netlink buffer size to 16MB.

/* Set socket buffer size to 3MB */
nl_socket_set_buffer_size(m_socket, 3145728, 0);

When there is a dump from kernel it is possible that netlink buffer run out of memory with 10k MAC scale. Hence this error is reported.

judyjoseph · 2022-11-23T16:27:06Z

Check if the CPU processing speed makes a difference
Can increase the netlink memory as a short term fix

dgsudharsan · 2022-12-07T21:32:06Z

@adyeung Can you please provide ETA for increasing netlink buffer size?

adyeung · 2022-12-29T22:58:16Z

Expecting a fix to be posted by 1/20/23

dgsudharsan · 2023-01-21T02:22:09Z

Expecting a fix to be posted by 1/20/23

@adyeung @kishorekunal01 Can you please share the fix if it is ready?

kishorekunal01 · 2023-01-23T19:15:08Z

I have created the pull request for the fix on 18th Jan sonic-net/sonic-swss-common#739 Thanks and Regards, Kishore Kunal

…

On Fri, Jan 20, 2023 at 6:22 PM Sudharsan Dhamal Gopalarathnam < ***@***.***> wrote: Expecting a fix to be posted by 1/20/23 @adyeung <https://github.com/adyeung> @kishorekunal01 <https://github.com/kishorekunal01> Can you please share the fix if it is ready? — Reply to this email directly, view it on GitHub <#12502 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APIREPFA25QJIGLPFHFQQRLWTNB5ZANCNFSM6AAAAAAROSSUFM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

dgsudharsan · 2023-02-22T17:01:39Z

Hi @kishorekunal01 We are still seeing the issue even after the fix. Should some parameters be adjusted?
root@qa-eth-vt03-2-3700v:~# show logging | grep fdb | grep ERR
Feb 22 18:13:46.561569 qa-eth-vt03-2-3700v ERR swss#fdbsyncd: :- readData: netlink reports out of memory on reading a netlink socket. High possibility of a lost message

dgsudharsan · 2023-02-22T17:02:45Z

sonic_dump_qa-eth-vt03-2-3700v_20230222_185508.tar.gz

I am attaching the techsupport here. You can see there are 10K MACS learnt in ASIC, However the kernel shows only 8K MACS

dgsudharsan · 2023-02-22T17:06:29Z

On a note someone had reported the same and increasing to 16 MB didn't help https://groups.google.com/g/sonicproject/c/Lc0cs-RzNSE

dgsudharsan · 2023-02-22T17:34:49Z

@kishorekunal01 Should we increase the netlink memory here? https://github.com/sonic-net/sonic-buildimage/blob/9ff2e2cff38fa71d0e5ce38f92d4339206849a74/files/image_config/sysctl/sysctl-net.conf. Currently it is 3MB
net.core.rmem_max=3145728
net.core.wmem_max=3145728

snider-nokia · 2023-03-02T20:36:23Z

@kishorekunal01 Should we increase the netlink memory here? https://github.com/sonic-net/sonic-buildimage/blob/9ff2e2cff38fa71d0e5ce38f92d4339206849a74/files/image_config/sysctl/sysctl-net.conf. Currently it is 3MB net.core.rmem_max=3145728 net.core.wmem_max=3145728

Yes, the net.core.rmem_max setting in the referenced file is the proper place to modify this setting. Please see my associated comment here at sonic-net/sonic-swss-common#739 (comment)

Issue #12587 relates to netlink messages originated at the kernel and sent to application (kernel-to-socket), thus only the socket rcv buffer needs increasing in the context of that issue.

If the problematic path here is instead from netlink socket-to-kernel as opposed to from kernel-to-socket then the net.core.wmem_max setting will need to be increased for the sake of larger socket tx buffer.

dgsudharsan added the Issue for 202205 label Oct 26, 2022

judyjoseph assigned adyeung Nov 23, 2022

judyjoseph added BRCM Triaged this issue has been triaged labels Nov 23, 2022

judyjoseph mentioned this issue Nov 23, 2022

[sonic-swss] - Netlink messages can be lost inbound from kernel and STATE_DB PORT_TABLE 'admin_status' field is misleading #12587

Open

adyeung assigned kishorekunal01 and unassigned adyeung Dec 29, 2022

dgsudharsan closed this as completed May 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FDB]All MACs are not synced to the kernel in scale scenario #12502

[FDB]All MACs are not synced to the kernel in scale scenario #12502

dgsudharsan commented Oct 26, 2022 •

edited

Loading

adyeung commented Oct 28, 2022

kishorekunal01 commented Oct 29, 2022 •

edited

Loading

kishorekunal01 commented Oct 29, 2022 •

edited

Loading

dgsudharsan commented Oct 31, 2022

kishorekunal01 commented Oct 31, 2022 •

edited

Loading

kishorekunal01 commented Oct 31, 2022

dgsudharsan commented Nov 3, 2022

kishorekunal01 commented Nov 15, 2022

judyjoseph commented Nov 23, 2022

dgsudharsan commented Dec 7, 2022

adyeung commented Dec 29, 2022

dgsudharsan commented Jan 21, 2023

kishorekunal01 commented Jan 23, 2023 via email

dgsudharsan commented Feb 22, 2023

dgsudharsan commented Feb 22, 2023

dgsudharsan commented Feb 22, 2023

dgsudharsan commented Feb 22, 2023

snider-nokia commented Mar 2, 2023 •

edited

Loading

[FDB]All MACs are not synced to the kernel in scale scenario #12502

[FDB]All MACs are not synced to the kernel in scale scenario #12502

Comments

dgsudharsan commented Oct 26, 2022 • edited Loading

Description

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

Output of show techsupport:

Additional information you deem important (e.g. issue happens only occasionally):

adyeung commented Oct 28, 2022

kishorekunal01 commented Oct 29, 2022 • edited Loading

kishorekunal01 commented Oct 29, 2022 • edited Loading

dgsudharsan commented Oct 31, 2022

kishorekunal01 commented Oct 31, 2022 • edited Loading

kishorekunal01 commented Oct 31, 2022

dgsudharsan commented Nov 3, 2022

kishorekunal01 commented Nov 15, 2022

judyjoseph commented Nov 23, 2022

dgsudharsan commented Dec 7, 2022

adyeung commented Dec 29, 2022

dgsudharsan commented Jan 21, 2023

kishorekunal01 commented Jan 23, 2023 via email

dgsudharsan commented Feb 22, 2023

dgsudharsan commented Feb 22, 2023

dgsudharsan commented Feb 22, 2023

dgsudharsan commented Feb 22, 2023

snider-nokia commented Mar 2, 2023 • edited Loading

dgsudharsan commented Oct 26, 2022 •

edited

Loading

Output of `show version`:

Output of `show techsupport`:

kishorekunal01 commented Oct 29, 2022 •

edited

Loading

kishorekunal01 commented Oct 29, 2022 •

edited

Loading

kishorekunal01 commented Oct 31, 2022 •

edited

Loading

snider-nokia commented Mar 2, 2023 •

edited

Loading