Skip to content

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pavelsavara opened this issue Jul 24, 2023 · 12 comments
Open

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

pavelsavara opened this issue Jul 24, 2023 · 12 comments
Assignees
@radekdoulik
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Milestone
10.0.0

Comments

@pavelsavara
Copy link
Member

pavelsavara commented Jul 24, 2023 •
edited by build-analysis bot
Loading

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": ["emcc", "SIGKILL"],
  "ExcludeConsoleLog": false
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707
Error message validated: [emcc SIGKILL]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/25/2024 7:29:32 AM UTC

Report

Build Definition Test Pull Request
1035601 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1035583 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115335
1035481 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115254
1035467 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115327
1034762 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115130
1035420 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115344
1035414 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115236
1035406 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115351
1035293 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115352
1035256 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114881
1035165 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115344
1035077 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114523
1035083 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115131
1035080 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113907
1034997 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
1035032 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115315
1034984 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1034868 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115342
1034856 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114881
1034834 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115341
1034698 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115335
1034819 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115327
1034810 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115062
1034774 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115131
1034770 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115311
1034759 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115113
1034717 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115308
1034344 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115257
1034449 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114148
1034591 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115330
1034516 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Options.ConfigurationExtensions.SourceGeneration.Tests.WorkItemExecution
1034490 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115327
1034061 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115290
1034471 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115158
1034482 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1032619 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115254
1034455 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution #114629
1034408 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114469
1034360 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115320
1034002 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115130
1034201 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115314
1034195 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115315
1034085 dotnet/runtime WasmTestOnChrome-ST-System.CodeDom.Tests.WorkItemExecution
1034110 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115235
1034055 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114881
1034049 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115313
1034026 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1033955 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115310
1033928 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115062
1033914 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Intrinsics.Tests.WorkItemExecution #114800
1033870 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115257
1033836 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115308
1033816 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115131
1033675 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114800
1033795 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114619
1033743 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114497
1033711 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115062
1033115 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115278
1033106 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115268
1033103 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1033062 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115275
1033029 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115276
1032998 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Extensions.Tests.WorkItemExecution
1033006 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115271
1032941 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115271
1032983 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114881
1032979 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115275
1032962 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115272
1032909 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111292
1032878 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114828
1032888 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Extensions.Tests.WorkItemExecution
1032875 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115268
1032821 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115062
1032733 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115257
1032566 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115249
1032648 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115062
1032624 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113907
1032603 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115234
1032569 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115131
1032542 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114800
1032524 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113956
1032447 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115248
1032421 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114469
1032402 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115236
1032379 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115210
1032350 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115247
1032333 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115131
1031787 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114800
1032180 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115234
1032176 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Serialization.Json.ReflectionOnly.Tests.WorkItemExecution
1032183 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #114629
1032002 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115214
1032150 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
1032071 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115229
1032052 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115236
1032030 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115211
1031990 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115235
1031847 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115233
1031850 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113907
1031813 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #115131
Displaying 100 of 757 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
27 106 757
@pavelsavara pavelsavara added arch-wasm WebAssembly architecture blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Codegen-AOT-mono wasm-aot-test WebAssembly AOT Test Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm labels Jul 24, 2023
@pavelsavara pavelsavara added this to the 8.0.0 milestone Jul 24, 2023
@ghost
Copy link

ghost commented Jul 24, 2023

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": "emcc.*SIGKILL",
  "ExcludeConsoleLog": true
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Author: pavelsavara
Assignees: -
Labels:

arch-wasm, blocking-clean-ci, area-Codegen-AOT-mono, wasm-aot-test, Known Build Error, os-browser
Milestone: 8.0.0

@pavelsavara pavelsavara mentioned this issue Jul 24, 2023
Merged
@lewing
Copy link
Member

lewing commented Jul 24, 2023

Is this it hitting the ulimit of the container?

@lewing lewing self-assigned this Jul 24, 2023
@radical
Copy link
Member

radical commented Jul 24, 2023 

[ 7455.672962] [ 2825] 1000 2825 158937 41558 684032 0 0 python3 
[ 7455.672964] [ 4160] 102 4160 67335 453 180224 0 0 rsyslogd 
[ 7455.672966] [ 4181] 0 4181 455864 16951 634880 0 0 amacoreagent 
[ 7455.672968] [ 4453] 0 4453 298556 3977 249856 0 0 auoms 
[ 7455.672969] [ 4495] 0 4495 215902 6240 204800 0 0 auomscollect 
[ 7455.672971] [ 4750] 102 4750 191707 14176 622592 0 0 mdsd 
[ 7455.672973] [ 4967] 0 4967 334103 3030 253952 0 0 azsecd 
[ 7455.672975] [ 5046] 0 5046 352600 4193 270336 0 0 azsecmond 
[ 7455.672977] [ 32457] 0 32457 317321 3577 258048 0 0 azsecd 
[ 7455.672979] [ 2226] 0 2226 376486 334278 2924544 0 0 clamscan 
[ 7455.672981] [ 22067] 0 22067 180189 824 114688 0 -998 containerd-shim 
[ 7455.672983] [ 22089] 1000 22089 1159 16 57344 0 0 helix_docker_wo 
[ 7455.672985] [ 22156] 1000 22156 1159 17 53248 0 0 execute.sh 
[ 7455.672987] [ 22212] 1000 22212 5047 87 81920 0 0 bash 
[ 7455.672989] [ 22215] 1000 22215 68517493 67031 1163264 0 0 dotnet 
[ 7455.672990] [ 23320] 1000 23320 1159 17 61440 0 0 sh 
[ 7455.672992] [ 23321] 1000 23321 34332 22818 335872 0 0 python3 
[ 7455.672994] [ 23345] 1000 23345 1420573 1386396 11264000 0 0 wasm-opt 
[ 7455.672995] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/a2a4080691301bb74e3b282eab0b6f638e9d6f1227e62967b7fd46654706e554,task=wasm-opt,pid=23345,uid=1000 
[ 7455.673018] Out of memory: Killed process 23345 (wasm-opt) total-vm:5682292kB, anon-rss:5545584kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:11000kB oom_score_adj:0

@lewing
Copy link
Member

lewing commented Jul 25, 2023

lets disable these tests in aot for now

@lewing lewing mentioned this issue Aug 13, 2023
Open
3 tasks
@lewing
Copy link
Member

lewing commented Aug 13, 2023

closting as a duplicate of dotnet/dnceng#450

@lewing lewing closed this as completed Aug 13, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 12, 2023
@pavelsavara
Copy link
Member Author

I see this again https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-99833-merge-af0760c542d34f95a8/normal-System.Runtime.Tests/1/console.f849dad3.log?helixlogtype=result

@pavelsavara pavelsavara reopened this Apr 2, 2024
@lewing
Copy link
Member

lewing commented Apr 24, 2024

Analysis claims there are recent failures but the links are not updated?

@pavelsavara
Copy link
Member Author

we run most of those problematic cases only in runtime-wasm pipeline. Which needs to be triggered manually.
I updated the filter

@lewing lewing removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 2, 2024
@dotnet dotnet unlocked this conversation May 27, 2024
@akoeplinger akoeplinger modified the milestones: 8.0.0, 9.0.0 May 27, 2024
@lewing lewing modified the milestones: 9.0.0, 10.0.0 Aug 15, 2024
@lewing
Copy link
Member

lewing commented Feb 12, 2025

all the recent failures I looked at are in wasm-opt which is known to have pretty degenerate memory behavior when linking with debug symbols. I think we should investigate what we can do here. @radekdoulik lets discuss offline

@lewing lewing assigned radekdoulik and unassigned lewing Feb 12, 2025
@pavelsavara pavelsavara mentioned this issue Mar 21, 2025
Merged
@lewing
Copy link
Member

lewing commented Mar 21, 2025

we hope to see this ramp down now after #113755

@pavelsavara
Copy link
Member Author

pavelsavara commented Mar 21, 2025 •
edited
Loading

Maybe we will need to go down to -O0, I think I saw SIGKILL after #113755 , now we are at -O2

@lewing lewing mentioned this issue Apr 17, 2025
Draft
@lewing
Copy link
Member

lewing commented Apr 18, 2025

all the recent failures I looked at are in wasm-opt which is known to have pretty degenerate memory behavior when linking with debug symbols. I think we should investigate what we can do here. @radekdoulik lets discuss offline

#114805 seems to indicate it does help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@radekdoulik radekdoulik

Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Projects
None yet
Milestone
10.0.0
Development

No branches or pull requests
5 participants
@radical @lewing @pavelsavara @akoeplinger @radekdoulik