Allow for proposed updates to be published in an isolated, staged manner

[mthalman]

As pointed out in #1371 (comment), it's important to be able to validate updates before they become "live" and used by release builds. But this requires that the proposed changes be applied in an isolated Dockerfile and still published to the registry so the resulting image can be referenced by a test build.

This kind of isolation can be done either with multiple branches (e.g. main and staging) or a single branch with distinct tags that separate out the staging Dockerfile from the production one.

[mthalman]

[Triage] Keeping the single branch and just using distinct Dockerfiles/tags for staged updates is probably the simplest solution for now. It requires no new features/infra to support that. It's basically just a policy with documentation. Key actions that should be taken with this approach:

1. The contributor should file an issue whenever an update needs to be staged. This issue will be used to track the new Dockerfile to ensure it gets cleaned up.
2. The contributor creates a PR that references the issue with the changes. The changes should include new staging Dockerfiles only (this may be just copies of existing Dockerfiles with edits made to them) and their corresponding manifest updates.
3. PR gets merged and images published.
4. The contributor tests the changes by referencing the newly published images in the consuming repo.
5. If the testing indicates the changes are acceptable, a new PR is created with those edits being applied to the existing (production) Dockerfiles and the staging Dockerfiles that were added are removed. The issue in step 1 is referenced in this PR. The issue is closed when the PR is merged.
6. If the testing indicates the changes are not acceptable and no further changes are going to be tested, a new PR is created to delete the staging Dockerfiles. The issue in step 1 is referenced in this PR. The issue is closed when the PR is merged.

This should all be outlined in the README for contributors to reference.

[lbussell]

What is the proposed naming convention for the temporary staging images?

Here's an idea. What if each image always had two floating tags. For example:

• azurelinux-3.0-net10.0-build-amd64
• azurelinux-3.0-net10.0-build-preview-amd64

During normal operations both tags point to the "stable" image. If we need changes that might be destabilizing, then the preview tag can move to the new image with the process that @mthalman described above. Then,

1. Iterate on the preview tag until it's working with the required repo(s).
2. Then, the stable floating tag can be moved to the new version of the image.
3. The old stable image is removed.

Repos can choose to always use the stable or staging version of the tag, or they could even have one leg per each tag. Swap preview with your adjective of choice - staging, nightly, etc.

[mthalman]

One thing I think we should allow for is for multiple staged Dockerfiles for a given "base" image. We're in that situation today with #1371 and dotnet/runtime#112885 (comment). So I'd like to allow for flexibility and not be limited to a single Dockerfile for staging changes.