`partisan_plumtree_broadcast` crashing

[mcesaro]

Likely related to #251, when nodes go down and up a few times (usually when debugging apps!), dynamic node names appear apparently randomly, and for some reason this leads to a crash of partisan_plumtree_broadcast:

=ERROR REPORT==== 25-Oct-2023::15:00:14.183849 ===
** Generic server partisan_plumtree_broadcast terminating 
** Last message in was {'$gen_cast',
                           {broadcast,
                               {{{erleans_pm,registry},
                                 {'pebble@max-a5',
                                     {<<"01HDH23JRRXK9RYQC8K43ETEV9">>,
                                      pebble}}},
                                [{{12,
                                   '05afd3aa-727e-11ee-b34c-1d7199d1e1ea@127.0.0.1'},
                                  2},
                                 {{12,
                                   '0781260e-731c-11ee-90ab-1d7199d1e1ea@127.0.0.1'},
                                  2},
                                 {{12,
                                   '0899bc32-7280-11ee-83b5-1d71807ddbb4@127.0.0.1'},
                                  4},
                                 {{12,
                                   '186b0fe8-728b-11ee-9cf8-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   '1a54b050-728d-11ee-954c-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   '2732a858-732e-11ee-bb16-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   '289e1bdc-731f-11ee-acda-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   '3546105a-728f-11ee-ba13-1d715db1de9a@127.0.0.1'},
                                  6},
                                 {{12,
                                   '39aaafe8-7280-11ee-abd0-1d718ad5fce5@127.0.0.1'},
                                  2},
                                 {{12,
                                   '3c7269f6-728e-11ee-85c0-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   '3ca0c1ea-7336-11ee-8948-1d71e0acc23e@127.0.0.1'},
                                  2},
                                 {{12,
                                   '57ab948a-727b-11ee-9704-1d71fcbba7d2@127.0.0.1'},
                                  6},
                                 {{12,
                                   '6029245a-731c-11ee-af13-1d7199d1e1ea@127.0.0.1'},
                                  2},
                                 {{12,
                                   '658ddad8-727e-11ee-b0b7-1d7199d1e1ea@127.0.0.1'},
                                  2},
                                 {{12,
                                   '6590fc84-731b-11ee-a665-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   '6d6255c6-727f-11ee-97e0-1d7199d1e1ea@127.0.0.1'},
                                  2},
                                 {{12,
                                   '75e9a468-728b-11ee-ba84-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   '82f47e8a-7317-11ee-9b9e-1d7199d1e1ea@127.0.0.1'},
                                  4},
                                 {{12,
                                   '857d6546-7280-11ee-a96a-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   '879959be-727d-11ee-9e01-1d714e48ce70@127.0.0.1'},
                                  2},
                                 {{12,
                                   '995ffa4a-731d-11ee-9bcb-1d7199d1e1ea@127.0.0.1'},
                                  2},
                                 {{12,
                                   'a115769e-731c-11ee-ab7c-1d7199d1e1ea@127.0.0.1'},
                                  2},
                                 {{12,
                                   'acb54a5e-727e-11ee-bc0d-1d71764ec79c@127.0.0.1'},
                                  2},
                                 {{12,
                                   'fac15018-728c-11ee-865d-1d71e0acc23e@127.0.0.1'},
                                  1}]},
                               {object,
                                   {[{{12,
                                       '05afd3aa-727e-11ee-b34c-1d7199d1e1ea@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '0781260e-731c-11ee-90ab-1d7199d1e1ea@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '0899bc32-7280-11ee-83b5-1d71807ddbb4@127.0.0.1'},
                                      4,[]},
                                     {{12,
                                       '186b0fe8-728b-11ee-9cf8-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '1a54b050-728d-11ee-954c-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '2732a858-732e-11ee-bb16-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '289e1bdc-731f-11ee-acda-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '3546105a-728f-11ee-ba13-1d715db1de9a@127.0.0.1'},
                                      6,[]},
                                     {{12,
                                       '39aaafe8-7280-11ee-abd0-1d718ad5fce5@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '3c7269f6-728e-11ee-85c0-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '3ca0c1ea-7336-11ee-8948-1d71e0acc23e@127.0.0.1'},
                                      2,
                                      [{'$deleted',{1698,238814,182073}}]},
                                     {{12,
                                       '57ab948a-727b-11ee-9704-1d71fcbba7d2@127.0.0.1'},
                                      6,[]},
                                     {{12,
                                       '6029245a-731c-11ee-af13-1d7199d1e1ea@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '658ddad8-727e-11ee-b0b7-1d7199d1e1ea@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '6590fc84-731b-11ee-a665-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '6d6255c6-727f-11ee-97e0-1d7199d1e1ea@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '75e9a468-728b-11ee-ba84-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '82f47e8a-7317-11ee-9b9e-1d7199d1e1ea@127.0.0.1'},
                                      4,[]},
                                     {{12,
                                       '857d6546-7280-11ee-a96a-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '879959be-727d-11ee-9e01-1d714e48ce70@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       '995ffa4a-731d-11ee-9bcb-1d7199d1e1ea@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       'a115769e-731c-11ee-ab7c-1d7199d1e1ea@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       'acb54a5e-727e-11ee-bc0d-1d71764ec79c@127.0.0.1'},
                                      2,[]},
                                     {{12,
                                       'fac15018-728c-11ee-865d-1d71e0acc23e@127.0.0.1'},
                                      1,[]}],
                                    []}},
                               plum_db,0,'pebble@max-a5','pebble@max-a5'}}
** When Server state == {state,['clockwork@max-a5','pebble@max-a5',
                                'test@max-a5'],
                               [],#{},#{},
                               [partisan_plumtree_backend],
                               [],
                               ['clockwork@max-a5','pebble@max-a5',
                                'ska@max-a5','test@max-a5'],
                               1000,10000}
** Reason for termination ==
** {'module could not be loaded',
       [{plum_db,merge,
            [{{{erleans_pm,registry},
               {'pebble@max-a5',{<<"01HDH23JRRXK9RYQC8K43ETEV9">>,pebble}}},
              [{{12,'05afd3aa-727e-11ee-b34c-1d7199d1e1ea@127.0.0.1'},2},
               {{12,'0781260e-731c-11ee-90ab-1d7199d1e1ea@127.0.0.1'},2},
               {{12,'0899bc32-7280-11ee-83b5-1d71807ddbb4@127.0.0.1'},4},
               {{12,'186b0fe8-728b-11ee-9cf8-1d71764ec79c@127.0.0.1'},2},
               {{12,'1a54b050-728d-11ee-954c-1d71764ec79c@127.0.0.1'},2},
               {{12,'2732a858-732e-11ee-bb16-1d71764ec79c@127.0.0.1'},2},
               {{12,'289e1bdc-731f-11ee-acda-1d71764ec79c@127.0.0.1'},2},
               {{12,'3546105a-728f-11ee-ba13-1d715db1de9a@127.0.0.1'},6},
               {{12,'39aaafe8-7280-11ee-abd0-1d718ad5fce5@127.0.0.1'},2},
               {{12,'3c7269f6-728e-11ee-85c0-1d71764ec79c@127.0.0.1'},2},
               {{12,'3ca0c1ea-7336-11ee-8948-1d71e0acc23e@127.0.0.1'},2},
               {{12,'57ab948a-727b-11ee-9704-1d71fcbba7d2@127.0.0.1'},6},
               {{12,'6029245a-731c-11ee-af13-1d7199d1e1ea@127.0.0.1'},2},
               {{12,'658ddad8-727e-11ee-b0b7-1d7199d1e1ea@127.0.0.1'},2},
               {{12,'6590fc84-731b-11ee-a665-1d71764ec79c@127.0.0.1'},2},
               {{12,'6d6255c6-727f-11ee-97e0-1d7199d1e1ea@127.0.0.1'},2},
               {{12,'75e9a468-728b-11ee-ba84-1d71764ec79c@127.0.0.1'},2},
               {{12,'82f47e8a-7317-11ee-9b9e-1d7199d1e1ea@127.0.0.1'},4},
               {{12,'857d6546-7280-11ee-a96a-1d71764ec79c@127.0.0.1'},2},
               {{12,'879959be-727d-11ee-9e01-1d714e48ce70@127.0.0.1'},2},
               {{12,'995ffa4a-731d-11ee-9bcb-1d7199d1e1ea@127.0.0.1'},2},
               {{12,'a115769e-731c-11ee-ab7c-1d7199d1e1ea@127.0.0.1'},2},
               {{12,'acb54a5e-727e-11ee-bc0d-1d71764ec79c@127.0.0.1'},2},
               {{12,'fac15018-728c-11ee-865d-1d71e0acc23e@127.0.0.1'},1}]},
             {object,
                 {[{{12,'05afd3aa-727e-11ee-b34c-1d7199d1e1ea@127.0.0.1'},
                    2,[]},
                   {{12,'0781260e-731c-11ee-90ab-1d7199d1e1ea@127.0.0.1'},
                    2,[]},
                   {{12,'0899bc32-7280-11ee-83b5-1d71807ddbb4@127.0.0.1'},
                    4,[]},
                   {{12,'186b0fe8-728b-11ee-9cf8-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'1a54b050-728d-11ee-954c-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'2732a858-732e-11ee-bb16-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'289e1bdc-731f-11ee-acda-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'3546105a-728f-11ee-ba13-1d715db1de9a@127.0.0.1'},
                    6,[]},
                   {{12,'39aaafe8-7280-11ee-abd0-1d718ad5fce5@127.0.0.1'},
                    2,[]},
                   {{12,'3c7269f6-728e-11ee-85c0-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'3ca0c1ea-7336-11ee-8948-1d71e0acc23e@127.0.0.1'},
                    2,
                    [{'$deleted',{1698,238814,182073}}]},
                   {{12,'57ab948a-727b-11ee-9704-1d71fcbba7d2@127.0.0.1'},
                    6,[]},
                   {{12,'6029245a-731c-11ee-af13-1d7199d1e1ea@127.0.0.1'},
                    2,[]},
                   {{12,'658ddad8-727e-11ee-b0b7-1d7199d1e1ea@127.0.0.1'},
                    2,[]},
                   {{12,'6590fc84-731b-11ee-a665-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'6d6255c6-727f-11ee-97e0-1d7199d1e1ea@127.0.0.1'},
                    2,[]},
                   {{12,'75e9a468-728b-11ee-ba84-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'82f47e8a-7317-11ee-9b9e-1d7199d1e1ea@127.0.0.1'},
                    4,[]},
                   {{12,'857d6546-7280-11ee-a96a-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'879959be-727d-11ee-9e01-1d714e48ce70@127.0.0.1'},
                    2,[]},
                   {{12,'995ffa4a-731d-11ee-9bcb-1d7199d1e1ea@127.0.0.1'},
                    2,[]},
                   {{12,'a115769e-731c-11ee-ab7c-1d7199d1e1ea@127.0.0.1'},
                    2,[]},
                   {{12,'acb54a5e-727e-11ee-bc0d-1d71764ec79c@127.0.0.1'},
                    2,[]},
                   {{12,'fac15018-728c-11ee-865d-1d71e0acc23e@127.0.0.1'},
                    1,[]}],
                  []}}],
            []},
        {partisan_plumtree_broadcast,handle_cast,2,
            [{file,
                 "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
             {line,549}]},
        {gen_server,try_handle_cast,3,[{file,"gen_server.erl"},{line,1103}]},
        {gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,1165}]},
        {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,241}]}]}

=CRASH REPORT==== 25-Oct-2023::15:00:14.192338 ===
  crasher:
    initial call: partisan_plumtree_broadcast:init/1
    pid: <0.18755.0>
    registered_name: partisan_plumtree_broadcast
    exception error: undefined function plum_db:merge/2
      in function  partisan_plumtree_broadcast:handle_cast/2 (/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl, line 549)
      in call from gen_server:try_handle_cast/3 (gen_server.erl, line 1103)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1165)
    ancestors: [partisan_sup,<0.1077.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1078.0>,<0.1086.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [213968340301438561|119831794556142432]}}]
    trap_exit: false
    status: running
    heap_size: 46422
    stack_size: 28
    reductions: 77660
  neighbours:

=SUPERVISOR REPORT==== 25-Oct-2023::15:00:14.192677 ===
    supervisor: {local,partisan_sup}
    errorContext: child_terminated
    reason: {undef,
                [{plum_db,merge,
                     [{{{erleans_pm,registry},
                        {'pebble@max-a5',
                            {<<"01HDH23JRRXK9RYQC8K43ETEV9">>,pebble}}},
                       [{{12,'05afd3aa-727e-11ee-b34c-1d7199d1e1ea@127.0.0.1'},
                         2},
                        {{12,'0781260e-731c-11ee-90ab-1d7199d1e1ea@127.0.0.1'},
                         2},
                        {{12,'0899bc32-7280-11ee-83b5-1d71807ddbb4@127.0.0.1'},
                         4},
                        {{12,'186b0fe8-728b-11ee-9cf8-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'1a54b050-728d-11ee-954c-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'2732a858-732e-11ee-bb16-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'289e1bdc-731f-11ee-acda-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'3546105a-728f-11ee-ba13-1d715db1de9a@127.0.0.1'},
                         6},
                        {{12,'39aaafe8-7280-11ee-abd0-1d718ad5fce5@127.0.0.1'},
                         2},
                        {{12,'3c7269f6-728e-11ee-85c0-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'3ca0c1ea-7336-11ee-8948-1d71e0acc23e@127.0.0.1'},
                         2},
                        {{12,'57ab948a-727b-11ee-9704-1d71fcbba7d2@127.0.0.1'},
                         6},
                        {{12,'6029245a-731c-11ee-af13-1d7199d1e1ea@127.0.0.1'},
                         2},
                        {{12,'658ddad8-727e-11ee-b0b7-1d7199d1e1ea@127.0.0.1'},
                         2},
                        {{12,'6590fc84-731b-11ee-a665-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'6d6255c6-727f-11ee-97e0-1d7199d1e1ea@127.0.0.1'},
                         2},
                        {{12,'75e9a468-728b-11ee-ba84-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'82f47e8a-7317-11ee-9b9e-1d7199d1e1ea@127.0.0.1'},
                         4},
                        {{12,'857d6546-7280-11ee-a96a-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'879959be-727d-11ee-9e01-1d714e48ce70@127.0.0.1'},
                         2},
                        {{12,'995ffa4a-731d-11ee-9bcb-1d7199d1e1ea@127.0.0.1'},
                         2},
                        {{12,'a115769e-731c-11ee-ab7c-1d7199d1e1ea@127.0.0.1'},
                         2},
                        {{12,'acb54a5e-727e-11ee-bc0d-1d71764ec79c@127.0.0.1'},
                         2},
                        {{12,'fac15018-728c-11ee-865d-1d71e0acc23e@127.0.0.1'},
                         1}]},
                      {object,
                          {[{{12,
                              '05afd3aa-727e-11ee-b34c-1d7199d1e1ea@127.0.0.1'},
                             2,[]},
                            {{12,
                              '0781260e-731c-11ee-90ab-1d7199d1e1ea@127.0.0.1'},
                             2,[]},
                            {{12,
                              '0899bc32-7280-11ee-83b5-1d71807ddbb4@127.0.0.1'},
                             4,[]},
                            {{12,
                              '186b0fe8-728b-11ee-9cf8-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              '1a54b050-728d-11ee-954c-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              '2732a858-732e-11ee-bb16-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              '289e1bdc-731f-11ee-acda-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              '3546105a-728f-11ee-ba13-1d715db1de9a@127.0.0.1'},
                             6,[]},
                            {{12,
                              '39aaafe8-7280-11ee-abd0-1d718ad5fce5@127.0.0.1'},
                             2,[]},
                            {{12,
                              '3c7269f6-728e-11ee-85c0-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              '3ca0c1ea-7336-11ee-8948-1d71e0acc23e@127.0.0.1'},
                             2,
                             [{'$deleted',{1698,238814,182073}}]},
                            {{12,
                              '57ab948a-727b-11ee-9704-1d71fcbba7d2@127.0.0.1'},
                             6,[]},
                            {{12,
                              '6029245a-731c-11ee-af13-1d7199d1e1ea@127.0.0.1'},
                             2,[]},
                            {{12,
                              '658ddad8-727e-11ee-b0b7-1d7199d1e1ea@127.0.0.1'},
                             2,[]},
                            {{12,
                              '6590fc84-731b-11ee-a665-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              '6d6255c6-727f-11ee-97e0-1d7199d1e1ea@127.0.0.1'},
                             2,[]},
                            {{12,
                              '75e9a468-728b-11ee-ba84-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              '82f47e8a-7317-11ee-9b9e-1d7199d1e1ea@127.0.0.1'},
                             4,[]},
                            {{12,
                              '857d6546-7280-11ee-a96a-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              '879959be-727d-11ee-9e01-1d714e48ce70@127.0.0.1'},
                             2,[]},
                            {{12,
                              '995ffa4a-731d-11ee-9bcb-1d7199d1e1ea@127.0.0.1'},
                             2,[]},
                            {{12,
                              'a115769e-731c-11ee-ab7c-1d7199d1e1ea@127.0.0.1'},
                             2,[]},
                            {{12,
                              'acb54a5e-727e-11ee-bc0d-1d71764ec79c@127.0.0.1'},
                             2,[]},
                            {{12,
                              'fac15018-728c-11ee-865d-1d71e0acc23e@127.0.0.1'},
                             1,[]}],
                           []}}],
                     []},
                 {partisan_plumtree_broadcast,handle_cast,2,
                     [{file,
                          "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
                      {line,549}]},
                 {gen_server,try_handle_cast,3,
                     [{file,"gen_server.erl"},{line,1103}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1165}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.18755.0>},
               {id,partisan_plumtree_broadcast},
               {mfargs,{partisan_plumtree_broadcast,start_link,[]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,5000},
               {child_type,worker}]

It seems that the plum_db:merge function is not available because the module is not loaded.

[aramallo]

@mcesaro I need to take a look at this to see whether the issue is in Partisan, PlumDB or Erleans (our fork).

[mcesaro]

@aramallo Hi,
it's happening also on non-erlans applications:

=CRASH REPORT==== 26-Oct-2023::08:28:08.676084 ===
  crasher:
    initial call: partisan_plumtree_broadcast:init/1
    pid: <0.18966.0>
    registered_name: partisan_plumtree_broadcast
    exception error: undefined function plum_db:merge/2
      in function  partisan_plumtree_broadcast:handle_cast/2 (/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl, line 549)
      in call from gen_server:try_handle_cast/3 (gen_server.erl, line 1103)
      in call from gen_server:handle_msg/6 (gen_server.erl, line 1165)
    ancestors: [partisan_sup,<0.1007.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1008.0>,<0.1019.0>]
    dictionary: [{rand_seed,{#{type => exsss,next => #Fun<rand.0.65977474>,
                                bits => 58,uniform => #Fun<rand.1.65977474>,
                                uniform_n => #Fun<rand.2.65977474>,
                                jump => #Fun<rand.3.65977474>},
                              [125007732914024471|257206006587947701]}}]
    trap_exit: false
    status: running
    heap_size: 610
    stack_size: 28
    reductions: 40368
  neighbours:

=SUPERVISOR REPORT==== 26-Oct-2023::08:28:08.676618 ===
    supervisor: {local,partisan_sup}
    errorContext: child_terminated
    reason: {undef,
                [{plum_db,merge,
                     [{{{erleans_pm,registry},
                        {'pebble@max-a5',
                            {<<"01HDNAMM2S470TFV43NFD2GMNE">>,pebble}}},
                       [{{12,'e8a18d8a-73c7-11ee-bbf0-1d718ad5fce5@max-a5'},
                         2}]},
                      {object,
                          {[{{12,
                              'e8a18d8a-73c7-11ee-bbf0-1d718ad5fce5@max-a5'},
                             2,
                             [{'$deleted',{1698,301688,672946}}]}],
                           []}}],
                     []},
                 {partisan_plumtree_broadcast,handle_cast,2,
                     [{file,
                          "/home/max/work/ska/_build/default/lib/partisan/src/partisan_plumtree_broadcast.erl"},
                      {line,549}]},
                 {gen_server,try_handle_cast,3,
                     [{file,"gen_server.erl"},{line,1103}]},
                 {gen_server,handle_msg,6,
                     [{file,"gen_server.erl"},{line,1165}]},
                 {proc_lib,init_p_do_apply,3,
                     [{file,"proc_lib.erl"},{line,241}]}]}
    offender: [{pid,<0.18966.0>},
               {id,partisan_plumtree_broadcast},
               {mfargs,{partisan_plumtree_broadcast,start_link,[]}},
               {restart_type,permanent},
               {significant,false},
               {shutdown,5000},
               {child_type,worker}]

I think I can trigger this behaviour in my laptop by suspending it on Wifi1 and waking it on Wifi2.
It seems that when there is a network change, partisan still try to use the last known network.
Admittely not a common use case, but still it might be a nuisance. A potential workaround would be to have a function/API to signal the network change and reset all the connections; otherwise, the effect is that the running nodes are stuck in the try-to-reconnect loop, but also that new nodes cannot join the cluster.

[aramallo]

@mcesaro Thanks for the extra info. However, notice that since plum_db is in your logs, you are not running just Partisan, or you running plum_db directly? Otherwise, you are still running erleans (plum_db is a dependency in our fork).

[mcesaro]

@aramallo Hi, my bad. One of the servers also includes erleans as a dependency. So, I re-checked the dependencies for all the applications; I'm not explictly using erleans in a distributed mode as only one server is currently running erleans, but the network swtich issue now is limited to the host running erleans with plum_db.