Settings.InjectTopLevelFallback race condition can cause failures

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.15
Which Akka.NET Modules? Akka.Cluster.Tools, others too presumably

Describe the bug

Consider the following method in the ClusterSingletonManagerSettings class:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Singleton/ClusterSingletonManagerSettings.cs

Lines 28 to 36 in 55644b1

	public static ClusterSingletonManagerSettings Create(ActorSystem system)
	{
	system.Settings.InjectTopLevelFallback(ClusterSingletonManager.DefaultConfig());
	var config = system.Settings.Config.GetConfig("akka.cluster.singleton");
	if (config.IsNullOrEmpty())
	throw ConfigurationException.NullOrEmptyConfig<ClusterSingletonManagerSettings>("akka.cluster.singleton");
	return Create(config).WithRemovalMargin(Cluster.Get(system).DowningProvider.DownRemovalMargin);

It uses system.Settings.InjectTopLevelFallback in order to make sure that its serialization definitions are available for Akka.Remote so all of the Cluster.Singleton-specific messages can be deserialized.

However, the way system.Settings.InjectTopLevelFallback works is inherently racy - multiple components all starting up at the same time might inject their configs all at once and we can end up "missing" a config injection, which causes the following line to fail occasionally:

akka.net/src/contrib/cluster/Akka.Cluster.Tools/Singleton/ClusterSingletonManagerSettings.cs

Lines 33 to 34 in 55644b1

	if (config.IsNullOrEmpty())
	throw ConfigurationException.NullOrEmptyConfig<ClusterSingletonManagerSettings>("akka.cluster.singleton");

Expected behavior

Plugins should be able to add their configurations to Akka.NET in a thread-safe, reliable manner that is also CTOR-friendly (no-Task).

Actual behavior

There is a small chance that the plugin configuration never gets properly registered and it crash - it's also possible for the default HOCON to get quite large as a result of many plugins multiplicatively registering their configuration values.

Additional context

We had talked about solving this problem in v1.4, when we attempted to replace our built-in HOCON implementation with https://github.com/akkadotnet/HOCON. The suggestion at the time was to have some means looking for known configuration sections and then staging them altogether inside the ActorSystem.

Given that #7246 is a huge goal for v1.6, any sort of assembly-scanning solution is off the table - what we probably need is something functionally similar to InjectTopLevelFallback now but with thread-safety.

[CodingPythonMan]

This issue looks interesting and seems unassigned.
If no one is working on this yet, I'd like to give it a try.

I recently investigated #2357, so this seems like a natural next step.

[Aaronontheweb]

By all means, please go ahead - however, be forewarned: the right steps to take here are to propose a design change first (bear in mind our contributor guidelines: https://getakka.net/community/contributing/index.html) since this impacts live systems. So it'd be a good idea to propose a design here and maybe do a small proof of concept to get some feedback first.

[CodingPythonMan]

Got it, thanks for the clarification!
I'll take some time to prepare a design proposal and will post it here for review before moving forward with a POC.

[CodingPythonMan]

Design Proposal: Thread-Safe Configuration Injection for Akka.NET Plugins

Background

Currently, Akka.NET plugins such as Akka.Cluster.Tools inject fallback configuration during startup using:

system.Settings.InjectTopLevelFallback(SomeConfig);

This ensures that necessary configuration values (e.g., for serialization or actor behavior) are available when a plugin is initialized.
For example, in ClusterSingletonManagerSettings.Create(...), the config for akka.cluster.singleton is injected at runtime.

The Problem

This approach is not thread-safe.
If multiple plugins initialize at the same time and call InjectTopLevelFallback(...), a race condition may occur.

This can result in:

• Config values not being registered in time
• Runtime errors like:

throw ConfigurationException.NullOrEmptyConfig("akka.cluster.singleton");

• Config bloat due to duplicate fallback registrations
• Unpredictable behavior during system startup

Goals

We need a mechanism that:

• Is thread-safe and avoids race conditions
• Does not use Task, async, or reflection (to support AOT)
• Preserves the constructor-friendly initialization pattern
• Allows plugin authors to register configs safely and modularly

Proposed Solution

Introduce a static utility called ConfigurationStaging for safe, deferred config injection.

public static class ConfigurationStaging
{
    public static void Register(string key, Func<Config> configProvider);
    public static void ApplyAll(Settings settings);
}

How it works:

• Register(...) stores a config provider under a unique key inside a ConcurrentDictionary.
• ApplyAll(...) is called once during ActorSystem startup and injects all registered configs via InjectTopLevelFallback(...).

Example Plugin Usage
In a plugin such as Akka.Cluster.Tools, config registration can be done like this:

static class ClusterSingletonManagerSettings
{
    static ClusterSingletonManagerSettings()
    {
        ConfigurationStaging.Register(
            "akka.cluster.singleton",
            () => ClusterSingletonManager.DefaultConfig()
        );
    }
}

This way, plugins do not inject config immediately. Instead, all configs are applied once in a controlled, deterministic order when the ActorSystem starts.