[FIXED] LeafNode: propagation of (no)interest issues #6161
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There were multiple issues, but basically the fact that we would not store the routed subscriptions with the origin of the LEAF they came from made the server unable to differentiate those compared to "local" routed subscriptions, which in some cases (like a server restart and the resend of subscriptions) could lead to servers sending incorrectly subscription interest to leaf connections. We are now storing the subscriptions with a sub type indicator and the origin (for leaf subscriptions) as part of the key. This allows to differentiate "regular" routed subs versus the ones on behalf of a leafnode. An INFO boolean is added `LNOCU` to indicate support for origin in the `LS-` protocol, which is required to properly handle the removal. Therefore, if a route does not have `LNOCU`, the server will behave like an old server, and store with the key that does not contain the origin, so that it can be removed when getting an LS- without the origin. Note that in the case of a mix of servers in the same cluster, some of the issues this PR is trying to fix will be present (since the server will basically behave like a server without the fix). Having a different routed subs for leaf connections allow to revisit the fix #5982 that was done for issue #5972, which was about a more fair queue distribution to a cluster of leaf connections. That fix actually introduced a change in that we always wanted to favor queue subscriptions of the cluster where the message is produced, which that fix possibly changed. With this current PR, the server can now know if a remote queue sub is for a "local" queue sub there or on behalf of a leaf and therefore will not favor that route compared to a leaf subscription that it may have directly attached. Resolves #5972 Resolves #6148 Signed-off-by: Ivan Kozlovic <[email protected]>
kozlovic
commented
Nov 22, 2024
kozlovic
commented
Nov 22, 2024
server/leafnode.go
Outdated
| // Helper function to build the key that prevents collisions between normal | ||
| // routed subscriptions and routed subscriptions on behalf of a leafnode. | ||
| // Keys will look like this: | ||
| // "R foo" -> plain routed sub on "foo" |
There was a problem hiding this comment.
Will fix that tomorrow.
Signed-off-by: Ivan Kozlovic <[email protected]>
neilalexander
added a commit
that referenced
this pull request
Nov 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There were multiple issues, but basically the fact that we would not store the routed subscriptions with the origin of the LEAF they came from made the server unable to differentiate those compared to "local" routed subscriptions, which in some cases (like a server restart and the resend of subscriptions) could lead to servers sending incorrectly subscription interest to leaf connections.
We are now storing the subscriptions with a sub type indicator and the origin (for leaf subscriptions) as part of the key. This allows to differentiate "regular" routed subs versus the ones on behalf of a leafnode.
An INFO boolean is added
LNOCUto indicate support for origin in theLS-protocol, which is required to properly handle the removal. Therefore, if a route does not haveLNOCU, the server will behave like an old server, and store with the key that does not contain the origin, so that it can be removed when getting an LS- without the origin. Note that in the case of a mix of servers in the same cluster, some of the issues this PR is trying to fix will be present (since the server will basically behave like a server without the fix).Having a different routed subs for leaf connections allow to revisit the fix #5982 that was done for issue #5972, which was about a more fair queue distribution to a cluster of leaf connections. That fix actually introduced a change in that we always wanted to favor queue subscriptions of the cluster where the message is produced, which that fix possibly changed. With this current PR, the server can now know if a remote queue sub is for a "local" queue sub there or on behalf of a leaf and therefore will not favor that route compared to a leaf subscription that it may have directly attached.
Resolves #5972
Resolves #6148
Signed-off-by: Ivan Kozlovic [email protected]