Repository navigation
Description
Description. With SocketsHttpHandler over HTTP/2, a request whose request content is still being sent (a duplex streaming request) receives its complete response, including the trailing HEADERS frame with END_STREAM. The server then sends RST_STREAM(NO_ERROR) for the stream (RFC 9113 section 8.1: a server may complete a response before the request and reset the stream with NO_ERROR), GOAWAY(NO_ERROR) with a last stream ID that includes the stream, and closes the TCP connection. Intermittently, the next read of the response content stream throws HttpProtocolException "The HTTP/2 server closed the connection. HTTP/2 error code 'NO_ERROR' (0x0)." instead of returning the remaining buffered data and then 0, and the trailing headers are not available.
Cause as read in the source (release/10.0). When Http2Connection.ProcessIncomingFramesAsync reads the end of the connection after a GOAWAY, it fails every stream still registered on the connection with the GOAWAY error code. A stream stays registered until both its request and its response are complete; a stream whose response is complete but whose request content has not completed is therefore failed, and CheckResponseBodyState then throws the stored exception on the next read although the response was fully received. Whether the application's read runs before or after the connection processes the end of the connection decides the outcome, so the failure rate rises with thread-pool load.
Reproduction. The console project below (Kestrel server, Grpc.Net.Client 2.80.0 as the client, .NET 10.0.14 on Linux): 17 of 200 runs without load and 54 of 200 runs under thread-pool load fail with the exception above; the rest read the trailers.
Expected. A stream whose response has been completely received (END_STREAM on the trailing HEADERS) delivers its response content and trailing headers even if the connection closes afterwards; only the request content of such a stream fails, or it is completed as reset with NO_ERROR.
Reproduction project
Repro.csproj:
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net10.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App" />
<PackageReference Include="Grpc.Net.Client" Version="2.80.0" />
</ItemGroup>
</Project>Program.cs:
using System.Net;
using System.Net.Sockets;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Server.Kestrel.Core;
// Usage: dotnet run -c Release -- <runs> [load]
var runs = args.Length > 0 ? int.Parse(args[0]) : 100;
var load = args.Length > 1 && args[1] == "load";
var bytes = Marshallers.Create<byte[]>(value => value, value => value);
var method = new Method<byte[], byte[]>(MethodType.DuplexStreaming, "repro.Service", "Connect", bytes, bytes);
var outcomes = new SortedDictionary<string, int>(StringComparer.Ordinal);
for (var run = 0; run < runs; run++)
{
var port = FreePort();
var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
var end = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
var builder = WebApplication.CreateEmptyBuilder(new WebApplicationOptions());
builder.WebHost.UseKestrelCore().ConfigureKestrel(kestrel => kestrel.Listen(IPAddress.Loopback, port, listen => listen.Protocols = HttpProtocols.Http2));
await using var app = builder.Build();
// The server answers the duplex call with response headers, then ends it with trailers (grpc-status 14) without reading the request body.
app.Run(async context =>
{
context.Response.ContentType = "application/grpc";
await context.Response.Body.FlushAsync();
started.TrySetResult();
await end.Task;
context.Response.AppendTrailer("grpc-status", "14");
context.Response.AppendTrailer("grpc-message", "closed by the server");
});
await app.StartAsync();
using var channel = GrpcChannel.ForAddress($"http://127.0.0.1:{port}");
using var call = channel.CreateCallInvoker().AsyncDuplexStreamingCall(method, null, new CallOptions());
await started.Task;
// The client reads the response stream; its request stream stays open.
var read = call.ResponseStream.MoveNext(CancellationToken.None);
if (load)
for (var item = 0; item < Environment.ProcessorCount * 4; item++)
ThreadPool.UnsafeQueueUserWorkItem(_ => { var until = DateTime.UtcNow.AddMilliseconds(300); while (DateTime.UtcNow < until) { } }, null);
// The server ends the call with its trailers and stops gracefully: GOAWAY(NO_ERROR), then it closes the TCP connection.
end.SetResult();
await app.StopAsync();
string outcome;
try
{
outcome = await read ? "message" : "end of stream";
}
catch (RpcException exception)
{
outcome = $"{exception.StatusCode}: {exception.Status.Detail}";
}
outcomes[outcome] = outcomes.GetValueOrDefault(outcome) + 1;
}
foreach (var (outcome, count) in outcomes) Console.WriteLine($"{count} of {runs}: {outcome}");
static int FreePort()
{
using var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
return ((IPEndPoint)listener.LocalEndpoint).Port;
}Run: dotnet build -c Release, then dotnet bin/Release/net10.0/Repro.dll 200 and dotnet bin/Release/net10.0/Repro.dll 200 load.
Observed on Linux (.NET 10.0.14, 2 processors): 17 of 200 runs without load and 54 of 200 under thread-pool load end with the exception; on Windows 11 (.NET runtime 10.0.13, SDK 10.0.402): 2 of 200 without load and 8 of 200 under load.
Activity
dotnet-policy-service commentedon Nov 2, 2026
Tagging subscribers to this area: @karelz, @dotnet/ncl
See info in area-owners.md if you want to be subscribed.
ManickaP commentedon Nov 3, 2026
This needs less info. This is obviously LLM generated report. We need human written summary, description of real-world scenario where you encountered it. Or, if this is based only on LLM analysis of source code, we need to know that as well to appropriately triage it.
wolfgangcolsman commentedon Nov 3, 2026
Fair enough. Yes, I used an AI agent to write the report, and sorry about the length.
How we ran into it: we have a service that hosts a gRPC endpoint on Kestrel, and other services connect to it with Grpc.Net.Client over long-lived duplex streams. When we shut that endpoint down, we end each call with Unavailable and a message in the trailers, then stop Kestrel. One of our CI tests checks that the client sees that status. Now and then the client gets Internal with "The HTTP/2 server closed the connection ... NO_ERROR" instead, and that happens more often when the machine is busy.
The repro in the issue uses only Kestrel and Grpc.Net.Client. The explanation of why it happens inside SocketsHttpHandler comes from the agent reading the source. I haven't confirmed it with a debugger or a packet capture.
We worked around it by keeping the connection open after the shutdown until the client closes it, so it's not blocking us. I also filed the gRPC side as grpc/grpc-dotnet#2786.
ManickaP commentedon Nov 3, 2026
I'm pretty positive that this is just a dupe of #135063.
Feel free to share your findings there.