[chassis] packet forwarding issue on Clearwater2ms

[arlakshm]

On some ports the packet forwarding on the clearwater2ms card doesn't work.
When we do ping from the peer device, on some port the ping fails. On these ports the packets are received or transmitted on the phy. The link is up on the ASIC and peer-device.

Some debugs we collected during debug session
lua(slice 0) > /serdes info
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| SerDes Params | Index |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| Param | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_adapt | 2 | 2 | 5 | 5 | 2 | 2 | 4 | 4 | 2 | 2 | 2 | 2 | 1 | 1 | 1 | 1 | 3 | 3 | 3 | 3 | 1 | 1 | 1 | 1 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_agcgain | 16 | 23 | 9 | 18 | 20 | 25 | 24 | 28 | 34 | 30 | 27 | 46 | 49 | 61 | 62 | 36 | 29 | 29 | 36 | 38 | 57 | 40 | 40 | 27 |
| | 31 | 31 | 31 | 23 | 31 | 31 | 31 | 31 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 | 5 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_channel_est | 1.64 | 1.61 | 1.66 | 1.5 | 1.74 | 1.7 | 1.7 | 1.64 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| rx_channel_hf | 28 | 28 | 29 | 30 | 27 | 27 | 27 | 28 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| rx_channel_of | 46 | 45 | 48 | 45 | 47 | 46 | 46 | 46 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_ctle | 6 | 6 | 6 | 4 | 6 | 6 | 3 | 6 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 |
| | 3 | 3 | 3 | 6 | 3 | 3 | 5 | 3 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 | 7 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_dac | 15 | 15 | 8 | 9 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 14 | 13 | 14 | 14 | 15 | 15 | 15 | 15 |
| rx_delta | -4 | -2 | 3 | 3 | -6 | -4 | 0 | -4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_dfe | -0 | -0 | 0.766 | 0.728 | -0 | -0 | -0 | -0 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 5 | 4 | 4 | 5 | 4 | 4 | 4 | 4 |
| | -0 | -0 | 0.22 | 0.241 | -0 | -0 | -0 | -0 | -4 | -4 | -4 | -4 | -4 | -4 | -4 | -4 | -6 | -8 | -6 | -7 | -4 | -4 | -4 | -4 |
| | 0 | 0 | 0.288 | 0.331 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 3 | 1 | 3 | 0 | 0 | 0 | 0 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_edge | 39321 | 39321 | 39321 | 39321 | 43690 | 43690 | 43690 | 39321 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 | 43690 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_eye_height | 215 | 206 | 211 | 206 | 198 | 198 | 215 | 206 | 735 | 654 | 639 | 682 | 723 | 777 | 715 | 678 | 689 | 678 | 693 | 688 | 798 | 681 | 713 | 737 |
| | 258 | 241 | 234 | 241 | 232 | 232 | 232 | 241 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | 223 | 206 | 211 | 208 | 206 | 198 | 206 | 215 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_f1over3 | 5 | 5 | 5 | 4 | 5 | 5 | 5 | 5 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 | 3 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_ffe_taps | 115 | 151 | 115 | 234 | 122 | 120 | 120 | 135 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 |
| | 94 | 63 | 55 | -6 | 19 | 77 | 75 | 122 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 |
| | 1 | 1 | 1 | 122 | 95 | 1 | 1 | 1 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 |
| | 3 | 1 | 5 | -165 | -4 | 2 | -26 | -24 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 |
| | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 |
| | 1 | 1 | 1 | 4 | 4 | 1 | 1 | 1 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 |
| | 136 | 136 | 136 | 139 | 136 | 136 | 136 | 136 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 | 204 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| rx_graycode | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| rx_input_coupling | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| rx_linklost | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 |
| rx_msblsb | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| rx_pll_cap | 27 | 27 | 27 | 27 | 27 | 27 | 27 | 27 | 31 | 31 | 31 | 31 | 31 | 32 | 31 | 31 | 31 | 31 | 31 | 31 | 31 | 31 | 31 | 32 |
| rx_pol | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| rx_ppm | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | -35 | -35 | -35 | -35 | -35 | -35 | -35 | -35 | -40 | -40 | -35 | -40 | -35 | -35 | -35 | -35 |
| rx_precoder | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| rx_readapt | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| rx_ready | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 |
| rx_signal_detect | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 |
| rx_skef_addcap | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 |
| rx_skef_degen | 4 | 2 | 4 | 2 | 4 | 4 | 1 | 3 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |
| rx_skef_en | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| rx_skef_gain | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 | 32531 |
| tx_graycode | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| tx_msblsb | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| tx_pll_cap | 27 | 27 | 27 | 27 | 27 | 27 | 27 | 27 | 31 | 31 | 31 | 32 | 32 | 32 | 31 | 31 | 31 | 31 | 31 | 32 | 32 | 32 | 32 | 32 |
| tx_pol | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| tx_precoder | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
| tx_taps | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| | -5 | -5 | -5 | -5 | -5 | -5 | -5 | -5 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 | -1 |
| | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 | 15 |
| | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 | -6 |
| | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+-------------------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+

crlua(slice 0)
crlua(slice 0) > /lane status
+---------+------+---------+----------+---------------+------------+
| Lane | Mode | Speed | LB Mode | Tx State | Rx State |
| | | | | | SD RDY ADP |
+---------+------+---------+----------+---------------+------------+
| A0(0) | PAM4 | 53.125G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A1(1) | PAM4 | 53.125G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A2(2) | PAM4 | 53.125G | DISABLED | TRAFFIC | 1 1 1 |
| A3(3) | PAM4 | 53.125G | DISABLED | TRAFFIC | 1 1 1 |
| A4(4) | PAM4 | 53.125G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A5(5) | PAM4 | 53.125G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A6(6) | PAM4 | 53.125G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A7(7) | PAM4 | 53.125G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A8(8) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A9(9) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A10(10) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| A11(11) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| B0(12) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| B1(13) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| B2(14) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| B3(15) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| B4(16) | NRZ | 25.781G | DISABLED | TRAFFIC | 1 1 1 |
| B5(17) | NRZ | 25.781G | DISABLED | TRAFFIC | 1 1 1 |
| B6(18) | NRZ | 25.781G | DISABLED | TRAFFIC | 1 1 1 |
| B7(19) | NRZ | 25.781G | DISABLED | TRAFFIC | 1 1 1 |
| B8(20) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| B9(21) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| B10(22) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
| B11(23) | NRZ | 25.781G | DISABLED | FORCE_DISABLE | 0 0 0 |
+---------+------+---------+----------+---------------+------------+

crlua(slice 0) >

[arlakshm]

@kenneth-arista created this issue to track the clearwater2ms issue

[kenneth-arista]

Quick update. A new Credo SAI release is forthcoming to address this issue.

[byu343]

A new version v0.9.3 has been released. It can fix this issue from my testing.

[kenneth-arista]

Closing this issue as v0.9.3 Credo SAI should address this issue.