The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2742719	dotnet-aspire	Build	Log
2742703	dotnet-aspire	Publish Build Assets	Log
1085560	dotnet/aspnetcore	Run build.cmd helix target	Log
2742673	dotnet-aspire	Checkout dotnet-aspire@ankj/cli-signing-all to s	Log
1085184	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117256
1085478	dotnet/runtime	Send to Helix	Log
1085519	dotnet/roslyn	Test	Log
1085447	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117273
1085282	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62560
1085286	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62561
1085513	dotnet/roslyn	Test	Log
1085305	dotnet/sdk	Run AoT Tests in Helix	Log	dotnet/sdk#49652
1085439	dotnet/runtime	Send tests to Helix (Unix)	Log
1085463	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62559
1085465	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62559
1085315	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#49638
1085308	dotnet/sdk	Run AoT Tests	Log	dotnet/sdk#49654
1085452	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#49417
1085306	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49653
1085437	dotnet/aspnetcore	Run build.cmd helix target	Log
1085404	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#117287
1085284	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62560
1085369	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#117286
1085335	dotnet/runtime	Send tests to Helix (Unix)	Log
1085392	dotnet/roslyn	Test	Log
1085320	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#117254
1085291	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#117284
1085382	dotnet/roslyn	Test	Log
1085388	dotnet/sdk	Run AoT Tests in Helix	Log	dotnet/sdk#49579
1085384	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49547
1085269	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#117282
1085375	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49548
1085256	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#117246
1085274	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#117283
1085346	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62552
1085347	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62552
1085339	dotnet/aspnetcore	Run build.cmd helix target	Log
1085336	dotnet/aspnetcore	Run build.cmd helix target	Log
1085337	dotnet/aspnetcore	Run build.cmd helix target	Log
1085234	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113299
1085259	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#117246
1085288	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#62561
1085304	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#49651
1085303	dotnet/sdk	Run Tests in Helix	Log	dotnet/sdk#49650
1085163	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#117277
1085125	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#115335
1085158	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#117276
1085145	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#117191
1085073	dotnet/runtime	Send to Helix	Log	dotnet/runtime#117268
1085161	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#117276
2742353	dotnet-performance	Send job to Helix (Unix)	Log
2742329	dotnet-dotnet	Build (azurelinuxX64MuslCrossContainer)	Log
1084860	dotnet/runtime	Send to Helix	Log
2742262	dotnet-dotnet	Run Tests (azurelinuxSourceBuildTestContainer)	Log
1084824	dotnet/runtime	AOT compile CoreCLR tests	Log
1084560	dotnet/runtime	Build product	Log	dotnet/runtime#117254
2742031	dotnet-performance	Send job to Helix (Unix)	Log
2742022	dotnet-aspire	Build native packages	Log
1084378	dotnet/runtime	Send to Helix	Log
2741972	dotnet-aspire	Build native packages	Log
2741931	dotnet-dotnet-buildtools-prereqs-docker	🛡 Guardian: Post Analysis	Log
1084310	dotnet/roslyn	Run Integration Tests	Log
2741891	dotnet-aspire	Build os-specific packages	Log
1084231	maestro-auth-test/build-result-analysis-test	Run Tests (Windows)	Log	maestro-auth-test/build-result-analysis-test#2328
2741870	dotnet-arcade-services	Restore dependencies	Log
1084163	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116433
2741715	dotnet-dotnet	Build (azurelinuxArm64CrossContainer)	Log
2741683	dotnet-performance	Send job to Helix (Unix)	Log
1083778	dotnet/runtime	Send to Helix	Log
2741542	dotnet-aspire	Build os-specific packages	Log
2741541	dotnet-aspire	Build os-specific packages	Log
2741519	dotnet-aspire	🛡 Guardian: CredScan (Source)	Log
2741501	dotnet-aspire	Run Template tests	Log
1083263	dotnet/runtime	Send to Helix	Log
2741376	dotnet-aspire	Start Network Isolation	Log
2741357	dotnet-aspire	🛡 Guardian: AntiMalware Scanner (Source)	Log
2741354	dotnet-aspire	Build	Log
2741349	dotnet-aspire	🛡 Guardian: AntiMalware Scanner (Source)	Log
2741317	dotnet-performance	Send job to Helix (Unix)	Log
2741300	dotnet-aspire	Build	Log
2741283	dotnet-aspire	Run Template tests	Log
2741281	dotnet-aspire	Agent Pool Validation	Log
1083361	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#78941
2741273	dotnet-aspire	Checkout dotnet-aspire@ankj/cli-signing to s	Log
1083336	dotnet/roslyn	Run Integration Tests	Log
2740903	dotnet-performance	Send job to Helix (Unix)	Log
1082688	dotnet/runtime	Send to Helix	Log
1082647	dotnet/runtime	Send to Helix	Log	dotnet/runtime#116907
1082652	dotnet/runtime	AOT compile CoreCLR tests	Log
1082234	dotnet/runtime	Send tests to Helix (Unix)	Log
2740630	dotnet-dotnet	Run Tests	Log
1082394	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79208
2740583	dotnet-performance	Send job to Helix (Unix)	Log
1082283	dotnet/runtime	Send to Helix	Log
2740442	dotnet-aspnetcore	🛡 Component Governance	Log
2740431	dotnet-aspnetcore	Build Installers	Log
1082238	dotnet/runtime	Disk Usage before Build	Log
1082109	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#79202
2740291	dotnet-runtime	Send job to Helix (Unix)	Log
2740288	dotnet-dotnet	🛡 Guardian: PoliCheck	Log
Displaying 100 of 846 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
55	152	846

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)