DistributedPubSub: add a wait-for-subscribers value to Publish commands to delay publishing / discarding until subscribers are available

[Aaronontheweb]

Is your feature request related to a problem? Please describe.

A common problem that occurs with DistributedPubSub is when both the publisher and the subscriber start at essentially the same time, but on different nodes, and it takes a moment for that subscription to replicate. You end up with an immediate DeadLetter per #7626 and the publisher is never notified that this happened unless the end-user implements their own explicit ACK-reply system from all subscribers.

This makes DistributedPubSub kind of unwieldy to use in cases that require a higher degree of consistency.

Describe the solution you'd like

I think it would be worth doing two things:

1. Create a new PublishWithAck message that sends a reply back to the publisher, letting them know that the message was sent to N nodes with subscribers. That way the publisher can get feedback on whether or not their message was actually delivered to someone without having to write infrastructure to do this themselves in the subscriber actors.
2. Create a new WaitForSubscribers value - we shouldn't let the user do stuff like specifying the number of subscribers because users will 100% interpret that to mean "the number of subscribers" and not the "number of nodes with subscribers," the latter being what we actually can measure. This should also have a timeout property to prevent memory build up and the DistributedPubSub should have a max-buffer-size setting for the number of "waiting" messages it will allow per-topic.

Describe alternatives you've considered

The alternatives are forcing the user to do all of this themselves via the queries we added to support #3663 - which is very high-boilerplate, janky, and generally not good.

[jasonshave]

@Aaronontheweb is this the ack-reply 'pattern' that would work in the meantime?

• Subscribe in the constructor (or lifecycle startup of the actor)
• Create a message handler to receive the SubscribeAck and put a behavior switch in the handler delegate (i.e. Become(Ready))
• Stash all messages until ack is processed and behavior is switched

This basically puts the actor into a pseudo-waiting state on the subscription confirmation before starting to process messages. I suppose it could have an impact on the performance of the actor as a consequence of waiting for confirmation + stashing depending on how long it takes to get the ack.

[Aaronontheweb]

@jasonshave you'd also want to make sure that you send an ACK message to the Sender each time you get a message that you believe was sent via DistributedPubSub - it's that portion you'd really need to make this workable at publish-time.