DistributedPubSub: DeadLetter logging when no subscribers looks identical to the Mediator being dead

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.41
Which Akka.NET Modules? Akka.Cluster.Tools

Describe the bug

When we see a log that looks like this:

[INFO][05/05/2025 20:07:56.430Z][Thread 0050][akka://Sys/system/distributedPubSubMediator] Message [MyMessage] wrapped in [$Publish] from [akka://Sys/system/sharding/EntityActor/3/00d3n0000008nbfuai__a40UD000000xY7JYAU#832702173] to [akka://Sys/system/distributedPubSubMediator#450399993] was not delivered. [6] dead letters encountered. If this is not an expected behavior then [akka://Sys/system/distributedPubSubMediator#450399993] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. Message content: MyMessage

It can come from one of two places:

1. The Mediator is well and truly dead, which is what this log indicates
2. Or more likely, the Mediator is alive and well but has no subscribers for the given topic, which is where I suspect a large number of these logs are coming from. Relevant code:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/PublishSubscribe/DistributedPubSubMediator.cs

Lines 496 to 500 in f67e123

	private void IgnoreOrSendToDeadLetters(object message)
	{
	if (_settings.SendToDeadLettersWhenNoSubscribers)
	Context.System.DeadLetters.Tell(new DeadLetter(message, Sender, Context.Self));
	}

We need to make it clearer which is which when this happens so users can diagnose these problems with accurate information on what the system was doing at the time.

[Aaronontheweb]

So I'm not crazy - I had thought that we tried to fix this issue before: #5350