Build system: need to automatically check API changes against plugins

[Aaronontheweb]

Problem

With the release of Akka.NET v1.4.30, we've inadvertently broken existing Akka.DI plugin support as a result of merging in https://github.com/akkadotnet/akka.net/pull/5416/files#r767775286 - and specifically I reviewed this section and thought that level of API change was fine, attaining to our "Practical SemVer" standard. Any actively used Akka.DI plugins can be updated to 1.4.30 and this issue will be fixed on their end, but we shouldn't tolerate surprises like this. It's beneath our quality standard.

Akka.NET project members are more interested in making changes to older APIs in order to create improvements, take advantage of newer .NET features like Span<T> and so forth. Those are all well and good, but we don't support the ability to do that sufficiently safely right now without applying a ridiculous standard like strict SemVer.

A separate but related issue, explicitly documenting Akka.NET's standards and processes for contributors, has been on my to-do list for a while and I'm scheduled to complete it by December 31s.

Solution

What I think we need is some more rigorous compatibility testing against a set of downstream components to ensure that when APIs do get changed they are:

1. Wire compatible
2. Binary compatible
3. Behaviorally compatible

How I propose we do this:

First and foremost, we should never do this automatically on every pull request as it'll be prohibitively time-expensive. But when a member of @akkadotnet/core sees a significant change to an API approval or a Protobuf definition, they should always manually kick off one of these jobs via a command on the PR thread.

We'll have to design some specifications to actually implement this. Conceptually, they're easy.

API Compat Specs

1. Code against current minor version of Akka.NET;
2. Take explicit dependencies on last stable version of some plugin (Akka.Logger.Serilog, Alpakka, Akka.Management) - this step is mandatory. Without taking a downstream dependency we can't test for upstream binary-compatibility breakages.
3. Upgrade Akka.NET to PR'd NuGet packages;
4. Run a set of integration tests orchestrated using XUnit, Docker.NET et al.
5. Transform the pass / fail / skip output into a Github comment for the full set of API compat specs.

Wire Format Compat Specs

1. Build a set of stock docker images against each stable release of Akka.NET, labeled with that version;
2. Form a cluster using previous stable release - execute handful of cluster operations (pre-conditions check);
3. Build new docker images using PR'd NuGet;
4. Roll out update into cluster for only 50% of nodes;
5. Execute same cluster operations + assertions (post-conditions check);
6. Transform the pass / fail / skip output into a Github comment for the full set of API compat specs.

I think we should prioritize this as a Q1, 2022 milestone.