• Macsec port update, results into memory corruption
• Macsec profile replace -> enableMacsec -> disableMacsec[ deletes key in macsec_ports ] -> accessing dangling reference data -> memory
  corruption -> crash in macsecmgrd

How to Reproduce ?

• While running sonic-mgmt macsec tests between( Sonic - cEOS ), we seen macsecmgrd crashing.
• Sonic-mgmt tests replacing a macsec profile with new profile while running the test using sonic-db-cli
  Ref : https://github.com/sonic-net/sonic-mgmt/blob/master/tests/common/macsec/macsec_config_helper.py#L119
• We captured the valgrind logs on macsecmgrd process while running macsec sonic-mgmt tests

is there any macsec test gaps ?

• No, current existing sonic-mgmt tests are able to reproduce the issue

Valgrind traces:

==35== Invalid write of size 2
==35== at 0x484B403: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==35== by 0x4AF3C5A: std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_assign(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==35== by 0x12345B: assign (basic_string.h:1571)
==35== by 0x12345B: operator= (basic_string.h:805)
==35== by 0x12345B: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:537)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)
==35== Address 0x5710c00 is 80 bytes inside a block of size 136 free'd
==35== at 0x484499B: operator delete(void*, unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==35== by 0x11D3F8: deallocate (new_allocator.h:158)
==35== by 0x11D3F8: deallocate (alloc_traits.h:496)
==35== by 0x11D3F8: _M_put_node (stl_tree.h:565)
==35== by 0x11D3F8: _M_drop_node (stl_tree.h:632)
==35== by 0x11D3F8: _M_erase_aux (stl_tree.h:2495)
==35== by 0x11D3F8: erase (stl_tree.h:1209)
==35== by 0x11D3F8: erase (stl_map.h:1086)
==35== by 0x11D3F8: swss::MACsecMgr::disableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:608)
==35== by 0x123404: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:529)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)
==35== Block was alloc'd at
==35== at 0x4841F2F: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==35== by 0x123109: allocate (new_allocator.h:137)
==35== by 0x123109: allocate (alloc_traits.h:464)
==35== by 0x123109: _M_get_node (stl_tree.h:561)
==35== by 0x123109: _M_create_node<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:611)
==35== by 0x123109: _Auto_node<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:1636)
==35== by 0x123109: _M_emplace_unique<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:2433)
==35== by 0x123109: emplace<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_map.h:606)
==35== by 0x123109: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:507)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)
==35==
==35== Invalid write of size 1
==35== at 0x484B433: memmove (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==35== by 0x4AF3C5A: std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_assign(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==35== by 0x12345B: assign (basic_string.h:1571)
==35== by 0x12345B: operator= (basic_string.h:805)
==35== by 0x12345B: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:537)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)
==35== Address 0x5710c06 is 86 bytes inside a block of size 136 free'd
==35== at 0x484499B: operator delete(void*, unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==35== by 0x11D3F8: deallocate (new_allocator.h:158)
==35== by 0x11D3F8: deallocate (alloc_traits.h:496)
==35== by 0x11D3F8: _M_put_node (stl_tree.h:565)
==35== by 0x11D3F8: _M_drop_node (stl_tree.h:632)
==35== by 0x11D3F8: _M_erase_aux (stl_tree.h:2495)
==35== by 0x11D3F8: erase (stl_tree.h:1209)
==35== by 0x11D3F8: erase (stl_map.h:1086)
==35== by 0x11D3F8: swss::MACsecMgr::disableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:608)
==35== by 0x123404: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:529)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)
==35== Block was alloc'd at
==35== at 0x4841F2F: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==35== by 0x123109: allocate (new_allocator.h:137)
==35== by 0x123109: allocate (alloc_traits.h:464)
==35== by 0x123109: _M_get_node (stl_tree.h:561)
==35== by 0x123109: _M_create_node<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:611)
==35== by 0x123109: _Auto_node<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:1636)
==35== by 0x123109: _M_emplace_unique<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:2433)
==35== by 0x123109: emplace<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_map.h:606)
==35== by 0x123109: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:507)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)
==35==
==35== Invalid read of size 8
==35== at 0x4AF3C5B: std::__cxx11::basic_string<char, std::char_traits, std::allocator >::_M_assign(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==35== by 0x12345B: assign (basic_string.h:1571)
==35== by 0x12345B: operator= (basic_string.h:805)
==35== by 0x12345B: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:537)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)
==35== Address 0x5710bf0 is 64 bytes inside a block of size 136 free'd
==35== at 0x484499B: operator delete(void*, unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==35== by 0x11D3F8: deallocate (new_allocator.h:158)
==35== by 0x11D3F8: deallocate (alloc_traits.h:496)
==35== by 0x11D3F8: _M_put_node (stl_tree.h:565)
==35== by 0x11D3F8: _M_drop_node (stl_tree.h:632)
==35== by 0x11D3F8: _M_erase_aux (stl_tree.h:2495)
==35== by 0x11D3F8: erase (stl_tree.h:1209)
==35== by 0x11D3F8: erase (stl_map.h:1086)
==35== by 0x11D3F8: swss::MACsecMgr::disableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:608)
==35== by 0x123404: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:529)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)
==35== Block was alloc'd at
==35== at 0x4841F2F: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==35== by 0x123109: allocate (new_allocator.h:137)
==35== by 0x123109: allocate (alloc_traits.h:464)
==35== by 0x123109: _M_get_node (stl_tree.h:561)
==35== by 0x123109: _M_create_node<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:611)
==35== by 0x123109: _Auto_node<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:1636)
==35== by 0x123109: _M_emplace_unique<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_tree.h:2433)
==35== by 0x123109: emplace<const std::piecewise_construct_t&, std::tuple<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::tuple<> > (stl_map.h:606)
==35== by 0x123109: swss::MACsecMgr::enableMACsec(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&) (macsecmgr.cpp:507)
==35== by 0x120335: swss::MACsecMgr::doTask(Consumer&) (macsecmgr.cpp:313)
==35== by 0x12CFAC: drain (orch.cpp:258)
==35== by 0x12CFAC: drain (orch.cpp:255)
==35== by 0x12CFAC: Consumer::execute() (orch.cpp:252)
==35== by 0x1176D3: main (macsecmgrd.cpp:98)