3.1 ASP.NET core app, with runtime 3.1.3, randomly crashing w3wp.exe #21477
Assignees
Labels
area-identity
Includes: Identity and providers
External
This is an issue in a component not contained in this repository. It is open for tracking purposes.
Comments
jmatheti
changed the title
jmatheti
changed the title
jmatheti
changed the title
jmatheti
changed the title
|
@Janidbest - thanks for the report! I am interested in looking at your dump file. Can you host the dump file somewhere and email me directly a link to access it? My email address is levib [at] microsoft.com. If you don't have a place to host it let me know and I can dig one up for you. |
|
@GrabYourPitchforks Thanks for looking into. I have emailed the link. |
pranavkm
added
area-identity
|
@HaoK parking this on your plate to resolve. |
|
Are you able to share the source code of your app @Janidbest ? |
|
@HaoK We can not share the source code. I will try to create a similar sample app for you, can you let me know your findings so I will ensure to have those interested services the DI and see if I could reproduce the issue before I share it with you. Since its now area-identity, FYI, we have configuration driven |
|
If you look at the callstack in the crash, there appears to be recursion in the cookie validation paths, |
|
@HaoK Thanks. We have a code from samesite (breakin change 3.0- 3.1), in the startup services. This is is the only cookie related change compared to previous build which is working. I have been trying with the sample app to reproduce the issue, without a success. Now that I know its related to the cookie validation, let me add this and try. |
|
@HaoK we have got this issue in another environment on every login, on every login page get request is crashing and the site going down as the app pool stopped. I appreciate if you share your direct contact to expedite. Weird dev environment is working fine, but not test environment. |
|
sure my github is my microsoft alias, so [email protected] |
|
Are you getting an endless redirect loop by any chance, now you have samesite changes in? Is it on all browser types? Edge, Safari and Chrome? |
|
@blowdart Yes, we have seen this issue on Firefox, Edge and Chrome. But not sure about the Safari. I think I found it. The Since this issue not coming from the dev servers, we are also thinking this may be in the code where race condition hits when the server is under performing, rest all environments have a better performances where we see this issue. |
|
I'll leave it to Hao to dig in, but from your description at least it's not same site :) |
|
Looks like the root issue has been identified, resolving as external for now, feel free to reactivate if this requires anything on our side |
HaoK
added
the
External
And so what was the root issue? |
|
From an offline email, the offending call that was causing the recursion was: I have found the root cause. It is coming from the Identity server framework call var |
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
We have an application migrated from 2.1 to 3.1 which is working fine but failing randomly in production and UAT able to reproduce using ZAP scan. The stdout files are empty. The ANCM diagnostic log file does not have much information about the issue. Application log does not have any errors or warnings when the crash happens.
I have the dump file to share but that's over 500MB .
To use kestrel, outofprocess hosting model, we are hitting the port collision issue with 502.3 ANCM startup failure with empty stdout and one warning in the event viewer -
failed to start process with commandline 'dotnet .\App.dll' at stage 'PostStartCheck', ErrorCode = '0x8027025a', assigned port 25442, retryCounter '0'.
###Exception
Event viewer -
Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a
Faulting module name: KERNEL32.DLL, version: 10.0.14393.2969, time stamp: 0x5ccd17c9
Exception code: 0xc00000fd
Fault offset: 0x000000000001183d
Faulting process id: 0x1c7c
Faulting application start time: 0x01d6192b72f91477
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\System32\KERNEL32.DLL
Report Id: 8fac1967-f339-40d6-97ad-41e1cc2dc473
ANCM diagnostic log
[aspnetcorev2.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 65008.. File Version: 13.1.20018.2. Description: IIS ASP.NET Core Module V2. Commit: c3acdca.
[aspnetcorev2.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\DCA.Auth.Server.dll' path: 'E:\App'
[aspnetcorev2.dll] Known dotnet.exe location: ''
[aspnetcorev2.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[aspnetcorev2.dll] Resolving absolute path to dotnet.exe from 'dotnet.exe'
[aspnetcorev2.dll] Invoking where.exe to find dotnet.exe
[aspnetcorev2.dll] where.exe invocation returned: 'C:\Program Files\dotnet\dotnet.exe
C:\Program Files (x86)\dotnet\dotnet.exe
'
[aspnetcorev2.dll] Current process bitness type detected as isX64=1
[aspnetcorev2.dll] Processing entry 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Binary type 6
[aspnetcorev2.dll] Found dotnet.exe via where.exe invocation at 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\3.1.2\hostfxr.dll'
[aspnetcorev2.dll] Converted argument '.\App.dll' to 'E:\App.\App.dll'
[aspnetcorev2.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\3.1.2\hostfxr.dll' arguments:
[aspnetcorev2.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2.dll] Argument[1] = 'E:\App.\App.dll'
[aspnetcorev2.dll] Loading hostfxr from location C:\Program Files\dotnet\host\fxr\3.1.2\hostfxr.dll
[aspnetcorev2.dll] Canceling standard stream pipe reader
[aspnetcorev2.dll] Loading request handler: 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\3.1.2\aspnetcorev2_inprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_inprocess.dll] Initializing logs for 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\3.1.2\aspnetcorev2_inprocess.dll'. Process Id: 65008.. File Version: 13.1.20018.2. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: c3acdca.
[aspnetcorev2_inprocess.dll] Waiting for initialization
[aspnetcorev2_inprocess.dll] Starting in-process worker thread
[aspnetcorev2_inprocess.dll] Resolving hostfxr parameters for application: 'dotnet' arguments: '.\App.dll' path: 'E:\App'
[aspnetcorev2_inprocess.dll] Known dotnet.exe location: 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] Process path 'dotnet.exe' is dotnet, treating application as portable
[aspnetcorev2_inprocess.dll] Resolving absolute path to hostfxr.dll from 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] hostfxr.dll located at 'C:\Program Files\dotnet\host\fxr\3.1.2\hostfxr.dll'
[aspnetcorev2_inprocess.dll] Converted argument '.\App.dll' to 'E:\App.\App.dll'
[aspnetcorev2_inprocess.dll] Parsed hostfxr options: dotnet location: 'C:\Program Files\dotnet\dotnet.exe' hostfxr path: 'C:\Program Files\dotnet\host\fxr\3.1.2\hostfxr.dll' arguments:
[aspnetcorev2_inprocess.dll] Argument[0] = 'C:\Program Files\dotnet\dotnet.exe'
[aspnetcorev2_inprocess.dll] Argument[1] = 'E:\App.\App.dll'
[aspnetcorev2_inprocess.dll] Setting environment variable ASPNETCORE_IIS_HTTPAUTH=anonymous;
[aspnetcorev2_inprocess.dll] Setting environment variable ASPNETCORE_IIS_PHYSICAL_PATH=E:\App
[aspnetcorev2_inprocess.dll] Loading hostfxr from location C:\Program Files\dotnet\host\fxr\3.1.2\hostfxr.dll
[aspnetcorev2_inprocess.dll] Initial Dll directory: '', current directory: 'c:\windows\system32\inetsrv'
[aspnetcorev2_inprocess.dll] Setting dll directory to c:\windows\system32\inetsrv
[aspnetcorev2_inprocess.dll] Setting current directory to E:\App
[aspnetcorev2_inprocess.dll] In-process callbacks set
[aspnetcorev2_inprocess.dll] Event Log: 'Application 'E:\App' started successfully.'
End Event Log Message.
[aspnetcorev2_inprocess.dll] Starting app_offline monitoring in application 'E:\App'
[aspnetcorev2_inprocess.dll] Adding request. Total Request Count 1
[aspnetcorev2_inprocess.dll] Adding request. Total Request Count 2
[aspnetcorev2_inprocess.dll] Adding request. Total Request Count 3
[aspnetcorev2_inprocess.dll] Starting file watcher thread
From dump debug - exception
at compareinfo.windows.cs ,line 143
Unhandled exception at 0x00007FFCD58E183D (kernel32.dll) in w3wp.exe.14184.dmp: 0xC00000FD: Stack overflow (parameters: 0x0000000000000001, 0x000000213BE736C8).
Describe the bug
Application is going down after the 5 failures, as per the IIS app pool rapid-fail protection configuration.
Application errors in event viewer - IIS worker process crash
From the dump file stack over flow exception (see below)
To Reproduce
We don't know the exact scenario but when doing ZAP scan (with insane strength), we are able to reproduce the issue on UAT server.
Further technical details
dotnet --info- Issue exists with 2.1.9 and 3.1.3 runtime installed, even after removing 2.1.9 runtime.The text was updated successfully, but these errors were encountered: