Unstaked testnet nodes are receiving an inconsistent number gossip messages

[gregcusack]

Problem

Unstaked nodes on testnet are receiving an inconsistent number of gossip messages. For example, below is a plot showing an unstaked testnet node receiving push messages and pull responses. Periodically, the node will receive a normal amount of push messages (the "spikes") and then suddenly receive very few push messages:

This variable ingress push message rate is also causing the node's view of active stake in gossip to vary significantly at runtime. Initial problem brought up here: https://discord.com/channels/428295358100013066/478692221441409024/1390162668991807600
^ this problem was brought up during testnet restart but occurs in steady state as well.

Proposed Solution

Not sure yet. Sounds like poor gossip message propagation. Need to figure out all that was changed in gossip between v2.2 and v2.3

[gregcusack]

seems kind of obvious but had to confirm. this is a cluster-wide issue. Spun up a node running v2.2 on testnet and another running v2.3 code on mainnet. v2.2 code on testnet experienced the same inconsistencies and v2.3 code on mainnet had no issues

[willhickey]

According the v2.3 release schedule we would be asking for 10% of mainnet-beta stake to adopt v2.3 this week. From your current understanding of this bug:

1. Is 10% safe, or should we hold off on calling for the voluntary upgrade?
2. Any guess how long it will take to understand & fix this bug in v2.3?

[gregcusack]

TLDR: I believe 10% is safe. I am not convinced this is an actual bug per se but instead a result of how stake is distributed on testnet vs. mainnet.

Testnet has 3.5% unstaked nodes
Mainnet has 82% unstaked nodes

We propagate messages based on stake using WeightedShuffle. With only 3.5% unstaked nodes, it's very unlikely an unstaked node is going to end up in the push active set of a staked node. On mainnet with 82% unstaked, it's much more likely an unstaked node is going to end up in the active set of a staked node.

Thanks to @alexpyattaev , I was able to spin up a 240 node mininet cluster (also reproduced with a 400 node cluster). I modified the gossip code to read in a stake file that has the stake distribution described above - 3.5% unstaked for testnet sim and 82% unstaked for mainnet sim.

The plot below shows the number of packets received by 2 staked nodes and 2 unstaked nodes running master. The values on the left are for the testnet sim and the values on the right are the mainnet sim.

Unstaked nodes running in the testnet sim see periods where they are receiving much less traffic than their staked peers (sharp drops in traffic). However in the mainnet sim, the traffic received by staked and unstaked nodes remain relatively the same.

I ran the same experiment with v2.2.16 (mainnet) nodes (plot below). One test with the testnet stake distribution (left) and one with the mainnet stake distribution (right). These results exhibit similar behavior to the test above. For the testnet stake distribution, unstaked nodes tend to fall out of the gossip tree more often than the mainnet stake distribution:

Based on these results, I believe it is safe to go ahead with 10% stake adopting v2.3. waiting for @alexpyattaev to consider as well

However, this is likely something that needs to get fixed in the future with gossip. Just because we have fewer unstaked nodes on testnet, doesn't mean they should be receiving significantly less traffic.

[alexpyattaev]

@willhickey, the evidence from Greg is compelling, I agree that we should be ok with 10% adoption. This is unlikely to affect staked nodes anyway, and we should be able to see issues if we keep a close eye on these metrics.

We will also consider patches to gossip routing to make sure this does not happen again.

[gregcusack]

oh one sec. let me update: i realized I accidentally plotted packets_sent_push_messages_count instead of packets_received_push_messages_count. See the proper plots below. Left is testnet distribution w/ 4% unstaked and right is mainnet distribution 82% unstaked:

~We can see unstaked nodes w/ testnet stake distribution receive significantly less traffic than unstaked nodes running with the mainnet distribution:

Here is the updated second plot with nodes running v2.2.16 (what's running on mainnet). With testnet stake distribution (left) and mainnet stake distribution (right):

EDIT: just updated plots inline above here: #6903 (comment)

[gregcusack]

Also, just to add some more data here....Below is the packets_received_push_messages_count for the unstaked canaries going back three months. This includes when these canaries were running v2.2 (the same version on mainnet). We did not see a significant drop in packets received from the v2.2 to the v2.3 upgrade.