SDK Container publish fails on transient HTTP download error

[MattKotsenas]

Describe the bug

It appears that the SDK container publish task is not resilient to transient HTTP / networking errors on download. There's already retry on push that I found here: https://github.com/dotnet/sdk-container-builds/pull/256/files

However, on download we pass the HTTP stream directly to copy, which doesn't appear to have any retries wrapped around it.

Impact

This is more impactful that it might seem because it fails the build. That means in CI the whole pipeline fails or needs retried.

Exceptions (if any)

Microsoft.NET.Build.Containers.targets(352,5): error MSB4018: The "CreateImageIndex" task failed unexpectedly. [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018: System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded) [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at System.Net.Http.HttpConnection.FillAsync(Boolean async) [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, Boolean async, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken) [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken) [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.Registry.DownloadBlobAsync(String repository, Descriptor descriptor, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/Registry/Registry.cs:line 417 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.DockerCli.WriteImageLayers(TarWriter writer, BuiltImage image, SourceImageReference sourceReference, Func`2 layerPathFunc, CancellationToken cancellationToken, JsonArray layerTarballPaths) in /_/src/Containers/Microsoft.NET.Build.Containers/LocalDaemons/DockerCli.cs:line 338 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.DockerCli.WriteOciImageToBlobs(TarWriter writer, BuiltImage image, SourceImageReference sourceReference, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/LocalDaemons/DockerCli.cs:line 492 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.DockerCli.WriteMultiArchOciImageToStreamAsync(MultiArchImage multiArchImage, SourceImageReference sourceReference, DestinationImageReference destinationReference, Stream imageStream, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/LocalDaemons/DockerCli.cs:line 515 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.LocalDaemons.ArchiveFileRegistry.LoadAsync[T](T image, SourceImageReference sourceReference, DestinationImageReference destinationReference, CancellationToken cancellationToken, Func`6 writeStreamFunc) in /_/src/Containers/Microsoft.NET.Build.Containers/LocalDaemons/ArchiveFileRegistry.cs:line 48 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.LocalDaemons.ArchiveFileRegistry.LoadAsync[T](T image, SourceImageReference sourceReference, DestinationImageReference destinationReference, CancellationToken cancellationToken, Func`6 writeStreamFunc) in /_/src/Containers/Microsoft.NET.Build.Containers/LocalDaemons/ArchiveFileRegistry.cs:line 48 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.LocalDaemons.ArchiveFileRegistry.LoadAsync(MultiArchImage multiArchImage, SourceImageReference sourceReference, DestinationImageReference destinationReference, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/LocalDaemons/ArchiveFileRegistry.cs:line 60 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.ImagePublisher.PushToLocalRegistryAsync[T](T image, SourceImageReference sourceImageReference, DestinationImageReference destinationImageReference, TaskLoggingHelper Log, Telemetry telemetry, CancellationToken cancellationToken, Func`5 loadFunc) in /_/src/Containers/Microsoft.NET.Build.Containers/ImagePublisher.cs:line 107 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.ImagePublisher.PublishImageAsync(MultiArchImage multiArchImage, SourceImageReference sourceImageReference, DestinationImageReference destinationImageReference, TaskLoggingHelper Log, Telemetry telemetry, CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/ImagePublisher.cs:line 63 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.Tasks.CreateImageIndex.ExecuteAsync(CancellationToken cancellationToken) in /_/src/Containers/Microsoft.NET.Build.Containers/Tasks/CreateImageIndex.cs:line 86 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.NET.Build.Containers.Tasks.CreateImageIndex.Execute() in /_/src/Containers/Microsoft.NET.Build.Containers/Tasks/CreateImageIndex.cs:line 32 [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Execute() [/__w/1/s/Sample.csproj]
Microsoft.NET.Build.Containers.targets(352,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(TaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/__w/1/s/Sample.csproj]

Further technical details

dotnet sdk 8.0.407 on Windows

[MattKotsenas]

/cc @surayya-MS, @baronfel