Skip to content

[DPB] Port state is down after breakout on a port with dependencies without force #6646

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
slutati1536 opened this issue Feb 2, 2021 · 1 comment · Fixed by sonic-net/sonic-utilities#1303
Closed

[DPB] Port state is down after breakout on a port with dependencies without force #6646

slutati1536 opened this issue Feb 2, 2021 · 1 comment · Fixed by sonic-net/sonic-utilities#1303
Labels
Port Breakout 🔱

Comments

@slutati1536
Copy link
Contributor

Description

configure vlan and port channel dependencies on port,
try to breakout the port without force, breakout will fail as expected.
following the configuration, multiple dockers fail on the switch and several ports(included the port on which the breakout was configured) are observed to be in downstate permanently.

Steps to reproduce the issue:

  1. show interfaces status
  2. sudo config portchannel add PortChannel0001
  3. sudo config portchannel member add PortChannel0001 Ethernet212
  4. sudo config vlan add 1536
  5. sudo config vlan member add 1536 Ethernet212
  6. show vlan brief
  7. show interfaces portchannel
  8. show interfaces status
  9. sudo config interface breakout Ethernet212 4x25G[10G,1G] -y -v
  10. show interfaces status
  11. docker ps -a
    12.show interfaces status

Describe the results you received:

admin@r-lionfish-16:~$ show interfaces status 
  Interface            Lanes    Speed    MTU    FEC    Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  ---------------  -------  -----  -----  -------  ------  ------  -------  ---------------  ----------
  Ethernet0                0      25G   9100    N/A     etp1  routed      up       up   SFP/SFP+/SFP28         N/A
  Ethernet4                4      25G   9100    N/A     etp2  routed      up       up   SFP/SFP+/SFP28         N/A
  Ethernet8                8      25G   9100    N/A     etp3  routed      up       up   SFP/SFP+/SFP28         N/A
 Ethernet12               12      25G   9100    N/A     etp4  routed      up       up   SFP/SFP+/SFP28         N/A
 Ethernet16               16      25G   9100    N/A     etp5  routed      up       up   SFP/SFP+/SFP28         N/A
 Ethernet20               20      25G   9100    N/A     etp6  routed      up       up   SFP/SFP+/SFP28         N/A
....
Ethernet212  212,213,214,215     100G   9100    N/A    etp54  routed      up       up  QSFP28 or later         N/A
Ethernet216  216,217,218,219     100G   9100    N/A    etp55  routed      up       up  QSFP28 or later         N/A
Ethernet220  220,221,222,223     100G   9100    N/A    etp56  routed      up       up   QSFP+ or later         N/A
Ethernet224  224,225,226,227     100G   9100    N/A    etp57  routed      up       up   QSFP+ or later         N/A
Ethernet228  228,229,230,231     100G   9100    N/A    etp58  routed      up       up  QSFP28 or later         N/A
Ethernet232  232,233,234,235     100G   9100    N/A    etp59  routed      up       up  QSFP28 or later         N/A
Ethernet236  236,237,238,239      25G   9100    N/A    etp60  routed      up       up   SFP/SFP+/SFP28         N/A
admin@r-lionfish-16:~$ 

admin@r-lionfish-16:~$ show vlan brief
+-----------+--------------+-------------+----------------+-----------------------+-------------+
|   VLAN ID | IP Address   | Ports       | Port Tagging   | DHCP Helper Address   | Proxy ARP   |
+===========+==============+=============+================+=======================+=============+
|      1536 |              | Ethernet212 | tagged         |                       | disabled    |
+-----------+--------------+-------------+----------------+-----------------------+-------------+

admin@r-lionfish-16:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
       S - selected, D - deselected, * - not synced
  No.  Team Dev         Protocol     Ports
-----  ---------------  -----------  -------
 0001  PortChannel0001  LACP(A)(Dw)  N/A

admin@r-lionfish-16:~$ show interfaces status
Traceback (most recent call last):
  File "/usr/local/bin/intfutil", line 521, in <module>
    main()
  File "/usr/local/bin/intfutil", line 513, in main
    interface_stat.display_intf_status()
  File "/usr/local/bin/intfutil", line 354, in display_intf_status
    self.get_intf_status()
  File "/usr/local/lib/python3.7/dist-packages/utilities_common/multi_asic.py", line 137, in wrapped_run_on_all_asics
    func(self,  *args, **kwargs)
  File "/usr/local/bin/intfutil", line 435, in get_intf_status
    self.portchannel_speed_dict = po_speed_dict(self.po_int_dict, self.db)
  File "/usr/local/bin/intfutil", line 249, in po_speed_dict
    interface_speed = '{}G'.format(interface_speed[:-3])
TypeError: 'NoneType' object is not subscriptable

admin@r-lionfish-16:~$ docker ps -a
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                      PORTS               NAMES
f521db589013        docker-snmp:latest                   "/usr/local/bin/supe…"   7 hours ago         Up 8 seconds                                    snmp
b4aace4ad417        docker-sonic-mgmt-framework:latest   "/usr/local/bin/supe…"   7 hours ago         Up 7 hours                                      mgmt-framework
679283d4f900        docker-sonic-telemetry:latest        "/usr/local/bin/supe…"   7 hours ago         Up 6 minutes                                    telemetry
ac88d359b3e6        docker-router-advertiser:latest      "/usr/bin/docker-ini…"   7 hours ago         Exited (0) 49 seconds ago                       radv
1eba1f6083da        docker-platform-monitor:latest       "/usr/bin/docker_ini…"   7 hours ago         Exited (0) 56 seconds ago                       pmon
57b491953d6a        docker-lldp:latest                   "/usr/bin/docker-lld…"   7 hours ago         Up 6 minutes                                    lldp
00a31c0698cd        docker-dhcp-relay:latest             "/usr/bin/docker_ini…"   7 hours ago         Exited (0) 48 seconds ago                       dhcp_relay
57878ace3f75        docker-syncd-mlnx:latest             "/usr/local/bin/supe…"   7 hours ago         Exited (0) 54 seconds ago                       syncd
f11516278201        docker-teamd:latest                  "/usr/local/bin/supe…"   7 hours ago         Exited (0) 42 seconds ago                       teamd
217ae2b27018        docker-orchagent:latest              "/usr/bin/docker-ini…"   7 hours ago         Up 10 seconds                                   swss
271680d0c51c        docker-fpm-frr:latest                "/usr/bin/docker_ini…"   7 hours ago         Up 7 minutes                                    bgp
cc8ab013a8d0        docker-database:latest               "/usr/local/bin/dock…"   7 hours ago         Up 7 hours                                      database

admin@r-lionfish-16:~$ show interfaces status
      Interface            Lanes    Speed    MTU    FEC    Alias             Vlan    Oper    Admin    Type    Asym PFC
---------------  ---------------  -------  -----  -----  -------  ---------------  ------  -------  ------  ----------
      Ethernet0                0      25G   9100    N/A     etp1           routed    down       up     N/A         N/A
      Ethernet4                4      25G   9100    N/A     etp2           routed      up       up     N/A         N/A
      Ethernet8                8      25G   9100    N/A     etp3           routed      up       up     N/A         N/A
     Ethernet12               12      25G   9100    N/A     etp4           routed      up       up     N/A         N/A
     Ethernet16               16      25G   9100    N/A     etp5           routed      up       up     N/A         N/A
     Ethernet20               20      25G   9100    N/A     etp6           routed      up       up     N/A         N/A
     Ethernet24               24      25G   9100    N/A     etp7           routed      up       up     N/A         N/A
     Ethernet28               28      25G   9100    N/A     etp8           routed      up       up     N/A         N/A
     Ethernet32               32      25G   9100    N/A     etp9           routed      up       up     N/A         N/A
     Ethernet36               36      25G   9100    N/A    etp10           routed      up       up     N/A         N/A
     Ethernet40               40      25G   9100    N/A    etp11           routed      up       up     N/A         N/A
     Ethernet44               44      25G   9100    N/A    etp12           routed      up       up     N/A         N/A
     Ethernet48               48      25G   9100    N/A    etp13           routed      up       up     N/A         N/A
     Ethernet52               52      25G   9100    N/A    etp14           routed      up       up     N/A         N/A
     Ethernet56               56      25G   9100    N/A    etp15           routed      up       up     N/A         N/A
     Ethernet60               60      25G   9100    N/A    etp16           routed      up       up     N/A         N/A
     Ethernet64               64      25G   9100    N/A    etp17           routed      up       up     N/A         N/A
     Ethernet68               68      25G   9100    N/A    etp18           routed      up       up     N/A         N/A
     Ethernet72               72      25G   9100    N/A    etp19           routed      up       up     N/A         N/A
     Ethernet76               76      25G   9100    N/A    etp20           routed      up       up     N/A         N/A
     Ethernet80               80      25G   9100    N/A    etp21           routed      up       up     N/A         N/A
     Ethernet84               84      25G   9100    N/A    etp22           routed      up       up     N/A         N/A
     Ethernet88               88      25G   9100    N/A    etp23           routed      up       up     N/A         N/A
     Ethernet92               92      25G   9100    N/A    etp24           routed      up       up     N/A         N/A
     Ethernet96               96      25G   9100    N/A    etp25           routed      up       up     N/A         N/A
    Ethernet100              100      25G   9100    N/A    etp26           routed      up       up     N/A         N/A
    Ethernet104              104      25G   9100    N/A    etp27           routed      up       up     N/A         N/A
    Ethernet108              108      25G   9100    N/A    etp28           routed      up       up     N/A         N/A
    Ethernet112              112      25G   9100    N/A    etp29           routed      up       up     N/A         N/A
    Ethernet116              116      25G   9100    N/A    etp30           routed    down       up     N/A         N/A
    Ethernet120              120      25G   9100    N/A    etp31           routed      up       up     N/A         N/A
    Ethernet124              124      25G   9100    N/A    etp32           routed      up       up     N/A         N/A
    Ethernet128              128      25G   9100    N/A    etp33           routed      up       up     N/A         N/A
    Ethernet132              132      25G   9100    N/A    etp34           routed      up       up     N/A         N/A
    Ethernet136              136      25G   9100    N/A    etp35           routed      up       up     N/A         N/A
    Ethernet140              140      25G   9100    N/A    etp36           routed      up       up     N/A         N/A
    Ethernet144              144      25G   9100    N/A    etp37           routed      up       up     N/A         N/A
    Ethernet148              148      25G   9100    N/A    etp38           routed      up       up     N/A         N/A
    Ethernet152              152      25G   9100    N/A    etp39           routed      up       up     N/A         N/A
    Ethernet156              156      25G   9100    N/A    etp40           routed      up       up     N/A         N/A
    Ethernet160              160      25G   9100    N/A    etp41           routed      up       up     N/A         N/A
    Ethernet164              164      25G   9100    N/A    etp42           routed      up       up     N/A         N/A
    Ethernet168              168      25G   9100    N/A    etp43           routed      up       up     N/A         N/A
    Ethernet172              172      25G   9100    N/A    etp44           routed      up       up     N/A         N/A
    Ethernet176              176      25G   9100    N/A    etp45           routed      up       up     N/A         N/A
    Ethernet180              180      25G   9100    N/A    etp46           routed      up       up     N/A         N/A
    Ethernet184              184      25G   9100    N/A    etp47           routed      up       up     N/A         N/A
    Ethernet188              188      25G   9100    N/A    etp48           routed      up       up     N/A         N/A
    Ethernet192  192,193,194,195      25G   9100    N/A    etp49           routed      up       up     N/A         N/A
    Ethernet196          196,197      50G   9100    N/A   etp50a           routed      up       up     N/A         N/A
    Ethernet198          198,199      50G   9100    N/A   etp50c           routed      up       up     N/A         N/A
    Ethernet200          200,201      50G   9100    N/A   etp51a           routed      up       up     N/A         N/A
    Ethernet202          202,203      50G   9100    N/A   etp51c           routed      up       up     N/A         N/A
    Ethernet204              204      25G   9100    N/A   etp52a           routed      up       up     N/A         N/A
    Ethernet205              205      25G   9100    N/A   etp52b           routed      up       up     N/A         N/A
    Ethernet206              206      25G   9100    N/A   etp52c           routed      up       up     N/A         N/A
    Ethernet207              207      25G   9100    N/A   etp52d           routed      up       up     N/A         N/A
    Ethernet208              208      25G   9100    N/A   etp53a           routed      up       up     N/A         N/A
    Ethernet209              209      25G   9100    N/A   etp53b           routed      up       up     N/A         N/A
    Ethernet210              210      25G   9100    N/A   etp53c           routed      up       up     N/A         N/A
    Ethernet211              211      25G   9100    N/A   etp53d           routed      up       up     N/A         N/A
    Ethernet212  212,213,214,215     100G   9100    N/A    etp54  PortChannel0001    down     down     N/A         N/A
    Ethernet216  216,217,218,219     100G   9100    N/A    etp55           routed    down       up     N/A         N/A
    Ethernet220  220,221,222,223     100G   9100    N/A    etp56           routed      up       up     N/A         N/A
    Ethernet224  224,225,226,227     100G   9100    N/A    etp57           routed      up       up     N/A         N/A
    Ethernet228  228,229,230,231     100G   9100    N/A    etp58           routed      up       up     N/A         N/A
    Ethernet232  232,233,234,235     100G   9100    N/A    etp59           routed      up       up     N/A         N/A
    Ethernet236  236,237,238,239      25G   9100    N/A    etp60           routed      up       up     N/A         N/A
PortChannel0001              N/A     100G   9100    N/A      N/A           routed    down       up     N/A         N/A

Describe the results you expected:

no docker should fail following breakout command,
all ports should remain in upstate following the breakout configuration.

Additional information you deem important (e.g. issue happens only occasionally):

**Output of `show version`:**
SONiC Software Version: SONiC.dpb_qa_image.0-23134a64
Distribution: Debian 10.7
Kernel: 4.19.0-9-2-amd64
Build commit: 23134a64
Build date: Fri Jan 15 08:52:11 UTC 2021
Built by: vadymh@r-build-sonic03

Platform: x86_64-mlnx_msn3420-r0
HwSKU: ACS-MSN3420
ASIC: mellanox
ASIC Count: 1
Serial Number: MT2019X13878
Uptime: 16:24:58 up  7:29,  2 users,  load average: 2.75, 1.24, 1.45

Docker images:
REPOSITORY                    TAG                       IMAGE ID            SIZE
docker-syncd-mlnx             dpb_qa_image.0-23134a64   72cd4f0bad6c        540MB
docker-syncd-mlnx             latest                    72cd4f0bad6c        540MB
docker-sflow                  dpb_qa_image.0-23134a64   070e94ad45b1        465MB
docker-sflow                  latest                    070e94ad45b1        465MB
docker-snmp                   dpb_qa_image.0-23134a64   4513d9bbede9        436MB
docker-snmp                   latest                    4513d9bbede9        436MB
docker-dhcp-relay             dpb_qa_image.0-23134a64   e9f74defdbed        402MB
docker-dhcp-relay             latest                    e9f74defdbed        402MB
docker-teamd                  dpb_qa_image.0-23134a64   4432208a6469        463MB
docker-teamd                  latest                    4432208a6469        463MB
docker-nat                    dpb_qa_image.0-23134a64   b0f25765961f        466MB
docker-nat                    latest                    b0f25765961f        466MB
docker-router-advertiser      dpb_qa_image.0-23134a64   875949f0ef94        395MB
docker-router-advertiser      latest                    875949f0ef94        395MB
docker-platform-monitor       dpb_qa_image.0-23134a64   77567d9ce67c        683MB
docker-platform-monitor       latest                    77567d9ce67c        683MB
docker-lldp                   dpb_qa_image.0-23134a64   9bba135a3864        435MB
docker-lldp                   latest                    9bba135a3864        435MB
docker-database               dpb_qa_image.0-23134a64   ae8be589a33c        394MB
docker-database               latest                    ae8be589a33c        394MB
docker-sonic-mgmt-framework   dpb_qa_image.0-23134a64   9f7fe89d1421        608MB
docker-sonic-mgmt-framework   latest                    9f7fe89d1421        608MB
docker-orchagent              dpb_qa_image.0-23134a64   5928b24d51e3        481MB
docker-orchagent              latest                    5928b24d51e3        481MB
docker-macsec                 dpb_qa_image.0-23134a64   b255f6b0ccf1        466MB
docker-macsec                 latest                    b255f6b0ccf1        466MB
docker-sonic-telemetry        dpb_qa_image.0-23134a64   4e502f1867fd        469MB
docker-sonic-telemetry        latest                    4e502f1867fd        469MB
docker-fpm-frr                dpb_qa_image.0-23134a64   684115a1f6cc        480MB
docker-fpm-frr                latest                    684115a1f6cc        480MB

Attach debug file sudo generate_dump:

breakout_log_issue_1.txt

start reproduce:
Tue 02 Feb 2021 03:59:09 PM UTC
end reproduce:
Tue 02 Feb 2021 04:04:03 PM UTC
@praveen-li praveen-li mentioned this issue Feb 4, 2021
Merged
@slutati1536
Copy link
Contributor Author

duplicate of #6631, I close the issue.

lguohan pushed a commit to sonic-net/sonic-utilities that referenced this issue May 12, 2021
[DPB]: Shut down interface before dynamic port breakout (#1303)
a089e53 
Changes:
-- Shutdown the interfaces after config validation while Dy Port Breakout.
-- Validate del ports before calling breakOutPorts API.

Signed-off-by: Praveen Chaudhary [email protected]

Fixes sonic-net/sonic-buildimage#6646, sonic-net/sonic-buildimage#6631,

Signed-off-by: Praveen Chaudhary [email protected]
gitsabari pushed a commit to gitsabari/sonic-utilities that referenced this issue Jun 15, 2021
@gitsabari
[DPB]: Shut down interface before dynamic port breakout (sonic-net#1303)
5eaa5ad 
Changes:
-- Shutdown the interfaces after config validation while Dy Port Breakout.
-- Validate del ports before calling breakOutPorts API.

Signed-off-by: Praveen Chaudhary [email protected]

Fixes sonic-net/sonic-buildimage#6646, sonic-net/sonic-buildimage#6631,

Signed-off-by: Praveen Chaudhary [email protected]
malletvapid23 added a commit to malletvapid23/Sonic-Utility that referenced this issue Aug 3, 2023
@malletvapid23
[DPB]: Shut down interface before dynamic port breakout (#1303)
d88ac50 
Changes:
-- Shutdown the interfaces after config validation while Dy Port Breakout.
-- Validate del ports before calling breakOutPorts API.

Signed-off-by: Praveen Chaudhary [email protected]

Fixes sonic-net/sonic-buildimage#6646, sonic-net/sonic-buildimage#6631,

Signed-off-by: Praveen Chaudhary [email protected]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Port Breakout 🔱
Projects
None yet
Milestone
No milestone
2 participants
@lguohan @slutati1536