Grpc.Net.Client.GrpcChannel raises a SocketException when disposed

[MahmudOnWeb]

What version of gRPC and what language are you using?

<PackageReference Include="Google.Protobuf" Version="3.30.1" />
<PackageReference Include="Grpc.Net.Client" Version="2.70.0" />
<PackageReference Include="Grpc.Tools" Version="2.71.0">

On service project:
PackageReference Include="Grpc.AspNetCore" Version="2.70.0"

Current gRPC Asp.Net template sets version 2.64.0. With it the same behavior is observed.

What operating system (Linux, Windows,...) and version?

Runtime Environment:
OS Name: Windows
OS Version: 10.0.19045
OS Platform: Windows
RID: win-x64

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

.NET SDK:
Version: 9.0.201
Commit: 071aaccdc2
Workload version: 9.0.200-manifests.a3a1a094
MSBuild version: 17.13.13+1c2026462

Microsoft Visual Studio Community 2022
Version 17.13.3
VisualStudio.17.Release/17.13.3+35828.75

What did you do?

I've noticed some issues with GrpcClient Channel disposal, not sure since when. The system is working fine in unary and bi-directional streaming calls, no delays, no lost messages. etc. I've noticed that if there are no messages send for 80 seconds, the exception first occurs. I guess some timeout happens and it tries to automatically close the channel. If the application is closed without disposing the client, an exception occurs on server side (as it can be expected :))

Easiest way to reproduce this:
Follow the "Tutorial: Create a gRPC client and server in ASP.NET Core" on Msdn page (https://learn.microsoft.com/en-us/aspnet/core/tutorials/grpc/grpc-start?view=aspnetcore-9.0&tabs=visual-studio).

Start the service project. Tune the ports so that it is comfortable for you in launchSettings.json

info: Microsoft.Hosting.Lifetime[14]
Now listening on: https://localhost:7012
info: Microsoft.Hosting.Lifetime[14]
Now listening on: http://localhost:7102
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
Content root path: D:\development\gRPC\GrpcGreeter\GrpcGreeter
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
Request starting HTTP/2 POST https://localhost:7012/greet.Greeter/SayHello - application/grpc -
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
Executing endpoint 'gRPC - /greet.Greeter/SayHello'
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'gRPC - /greet.Greeter/SayHello'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished HTTP/2 POST https://localhost:7012/greet.Greeter/SayHello - 200 - application/grpc 84.3731ms

For the client "Do not use top-level statements" is set. The console code is as follows, basically copy/paste from tutorial with minor changes:

using Grpc.Net.Client;

namespace GrpcGreeterClient
{
internal class Program
{
static async Task Main(string[] args)
{
Console.WriteLine("Hello, World!");

        using(GrpcChannel channel = GrpcChannel.ForAddress("https://localhost:7012"))
        {
            Greeter.GreeterClient client = new Greeter.GreeterClient(channel);
            HelloReply reply = await client.SayHelloAsync(new HelloRequest { Name = "This is hello request." });

            Console.WriteLine(string.Format("Received: {0}", reply.Message));
            Console.WriteLine("Press ENTER to stop the demo");
            Console.ReadLine();
        }
    }
}

}

Run the client. It executes, service responds and it seems fine.

Hello, World!
Received: Hello This is hello request.
Press ENTER to stop the demo

However, in output window there are exception messages:

Exception thrown: 'System.IO.IOException' in System.Net.Sockets.dll
Exception thrown: 'System.IO.IOException' in System.Private.CoreLib.dll
Exception thrown: 'System.IO.IOException' in System.Private.CoreLib.dll
The program '[12348] GrpcGreeterClient.exe' has exited with code 0 (0x0).

From Visual Studio -> Debug -> Windows -> Exception settings click on Common Language Runtime Exception so that all exception are selected, there should be a clear tick sign, it should not be partially checked.

What did you expect to see?

I would expect that No SocketExceptions are raised and the channel disposes with no issues on application shutdown.
I would expect that No SocketExceptions are raised ~80 seconds after last gRPC call has been made.

What did you see instead?

Few exceptions have been raised. The call stack is as follows:

System.Net.Sockets.dll!System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(System.Net.Sockets.SocketError error, System.Threading.CancellationToken cancellationToken) Line 405 C#
System.Net.Sockets.dll!System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(short token) Line 308 C#
Grpc.Net.Client.dll!Grpc.Net.Client.Balancer.Internal.StreamWrapper.ReadAsync(System.Memory buffer, System.Threading.CancellationToken cancellationToken) Unknown
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBox<Grpc.Net.Client.Balancer.Internal.StreamWrapper.d__31>.ExecutionContextCallback(object s) Unknown
System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBox<Grpc.Net.Client.Balancer.Internal.StreamWrapper.d__31>.MoveNext(System.Threading.Thread threadPoolThread) Unknown
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncTaskMethodBuilder.AsyncStateMachineBox<Grpc.Net.Client.Balancer.Internal.StreamWrapper.d__31>.MoveNext() Unknown
System.Net.Sockets.dll!System.Net.Sockets.SocketAsyncEventArgs..cctor.AnonymousMethod__174_0(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* nativeOverlapped) Line 182 C#
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.IOCompletionPoller.Callback.Invoke(System.Threading.PortableThreadPool.IOCompletionPoller.Event e) Unknown
System.Private.CoreLib.dll!System.Threading.ThreadPoolTypedWorkItemQueue<System.Threading.PortableThreadPool.IOCompletionPoller.Event, System.Threading.PortableThreadPool.IOCompletionPoller.Callback>.System.Threading.IThreadPoolWorkItem.Execute() Unknown
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Unknown
System.Private.CoreLib.dll!System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() Unknown

Those are copy details from the exception:

System.IO.IOException
HResult=0x80131620
Message=Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..
Source=System.Net.Sockets
StackTrace:
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) in System.Net.Sockets\Socket.cs:line 405

Inner Exception 1:
SocketException: The I/O operation has been aborted because of either a thread exit or an application request.

Anything else we should know about your project / environment?

I've tired to reproduce the issue with tutorial project and (somehow) common environment.

[ricsiLT]

That timing feels familiar. Are you running on public cloud or private cloud? Is there any middleware that could kill idle connections? Could you reconfigure channel to have keepalives enabled and see if that helps?

[MahmudOnWeb]

That timing feels familiar. Are you running on public cloud or private cloud? Is there any middleware that could kill idle connections? Could you reconfigure channel to have keepalives enabled and see if that helps?

Thank you for your comment. This timing "thing" has been mentioned in few other issues. I believe a socked disposal exception or related causes the "timing" issue. After 80 or 100 seconds an automatic attempt to close (and maybe dispose) the channel is probably made, and that's why people are seeing this issue only after 80 or so seconds of inactivity.

Can you reproduce the tutorial example and get the same socket exception on your local machine? I've placed the link to msdn tutorial in my previous post