Akka.Cluster.Sharding: dealing with remember-entities and actors who can't start up correctly

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.41
Which Akka.NET Modules? Akka.Cluster.Sharding

Describe the bug

So an interesting scenario:

1. Actor gets created via Akka.Cluster.Sharding, but is misconfigured (i.e. missing DI registration that can't be resolved when the resolver.Props gets fired up) - crashes immediately;
2. Actor hits its "10 crashes within 1s and die" limit on the SupervisionStrategy, gets killed off by the Shard
3. Shard notices that the remembered-entity actor is dead, re-creates it; and
4. The cycle repeats, creating millions of errors until human intervention arrives.

Expected behavior

IMHO, the remember-entities system should not bypass the supervision system - if the supervisor recommends perma-death to a repeatedly failing actor, we need some way of honoring that and un-remembering the actor.

Actual behavior

Infinite churn between the remember-entities system and the restarting actor.