Description
Observed behavior
For a request/Reply service running on a leaf node connected to a supercluster replies would fail at the leaf node connection level (permission failure) unless GR.> was explicitely allowed.
Also, when the reply subject in question was used (in a messages) more than 2s after the subcription was made the issue disappeared.
In addition to being an itnernal optimization the effect confusing as the GR. prefix is conditionally applied. Sometimes its there, sometimes not (see details below).
Expected behavior
GR prefixing (routed replies) is an internal optimization, which should not be visible to client connections. AND leaf node connections (which should be treated as client connections).
A quick check in the server code showed that the GR prefix is conditionally applied when a matching subscription is in the account subscription cache (which times out after a few seconds). Both GR and the account subscription cache are server internal details.
Server and client version
2.11.3 and 2.10.29 (and earlier)
Host environment
Linux and Windows
Steps to reproduce
- Create a topology as per the diagram(US NAT and EU NATS being a supercluster).
- Set permissions on the leaf node connections restricted to the request and reply subjects used.
- The service is a boilerplate R/R server (reply to reply subject)
- The client subscribes to the reply subject. Waits for a configurable period of time. Sends the request.
- Service replies will be silently suppressed by the leaf node.
- Check the leaf node in debug mode and find: "Not permitted to deliver to GR......"
- When the client waits for 2+ seconds before sending the request the reply goes through (no GR prefixing )
- Disabling the account subscriber cache should also make the effect disappear