Skip to content

HTTP/2: a response that was fully received is failed with HttpProtocolException (NO_ERROR) when the server sends GOAWAY and closes the connection while the request body is still open #136834

Closed as duplicate

Description

@wolfgangcolsman

Description. With SocketsHttpHandler over HTTP/2, a request whose request content is still being sent (a duplex streaming request) receives its complete response, including the trailing HEADERS frame with END_STREAM. The server then sends RST_STREAM(NO_ERROR) for the stream (RFC 9113 section 8.1: a server may complete a response before the request and reset the stream with NO_ERROR), GOAWAY(NO_ERROR) with a last stream ID that includes the stream, and closes the TCP connection. Intermittently, the next read of the response content stream throws HttpProtocolException "The HTTP/2 server closed the connection. HTTP/2 error code 'NO_ERROR' (0x0)." instead of returning the remaining buffered data and then 0, and the trailing headers are not available.

Cause as read in the source (release/10.0). When Http2Connection.ProcessIncomingFramesAsync reads the end of the connection after a GOAWAY, it fails every stream still registered on the connection with the GOAWAY error code. A stream stays registered until both its request and its response are complete; a stream whose response is complete but whose request content has not completed is therefore failed, and CheckResponseBodyState then throws the stored exception on the next read although the response was fully received. Whether the application's read runs before or after the connection processes the end of the connection decides the outcome, so the failure rate rises with thread-pool load.

Reproduction. The console project below (Kestrel server, Grpc.Net.Client 2.80.0 as the client, .NET 10.0.14 on Linux): 17 of 200 runs without load and 54 of 200 runs under thread-pool load fail with the exception above; the rest read the trailers.

Expected. A stream whose response has been completely received (END_STREAM on the trailing HEADERS) delivers its response content and trailing headers even if the connection closes afterwards; only the request content of such a stream fails, or it is completed as reset with NO_ERROR.

Reproduction project

Repro.csproj:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup>
    <FrameworkReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Grpc.Net.Client" Version="2.80.0" />
  </ItemGroup>
</Project>

Program.cs:

using System.Net;
using System.Net.Sockets;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Server.Kestrel.Core;

// Usage: dotnet run -c Release -- <runs> [load]
var runs = args.Length > 0 ? int.Parse(args[0]) : 100;
var load = args.Length > 1 && args[1] == "load";
var bytes = Marshallers.Create<byte[]>(value => value, value => value);
var method = new Method<byte[], byte[]>(MethodType.DuplexStreaming, "repro.Service", "Connect", bytes, bytes);
var outcomes = new SortedDictionary<string, int>(StringComparer.Ordinal);
for (var run = 0; run < runs; run++)
{
	var port = FreePort();
	var started = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
	var end = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
	var builder = WebApplication.CreateEmptyBuilder(new WebApplicationOptions());
	builder.WebHost.UseKestrelCore().ConfigureKestrel(kestrel => kestrel.Listen(IPAddress.Loopback, port, listen => listen.Protocols = HttpProtocols.Http2));
	await using var app = builder.Build();
	// The server answers the duplex call with response headers, then ends it with trailers (grpc-status 14) without reading the request body.
	app.Run(async context =>
	{
		context.Response.ContentType = "application/grpc";
		await context.Response.Body.FlushAsync();
		started.TrySetResult();
		await end.Task;
		context.Response.AppendTrailer("grpc-status", "14");
		context.Response.AppendTrailer("grpc-message", "closed by the server");
	});
	await app.StartAsync();
	using var channel = GrpcChannel.ForAddress($"http://127.0.0.1:{port}");
	using var call = channel.CreateCallInvoker().AsyncDuplexStreamingCall(method, null, new CallOptions());
	await started.Task;
	// The client reads the response stream; its request stream stays open.
	var read = call.ResponseStream.MoveNext(CancellationToken.None);
	if (load)
		for (var item = 0; item < Environment.ProcessorCount * 4; item++)
			ThreadPool.UnsafeQueueUserWorkItem(_ => { var until = DateTime.UtcNow.AddMilliseconds(300); while (DateTime.UtcNow < until) { } }, null);
	// The server ends the call with its trailers and stops gracefully: GOAWAY(NO_ERROR), then it closes the TCP connection.
	end.SetResult();
	await app.StopAsync();
	string outcome;
	try
	{
		outcome = await read ? "message" : "end of stream";
	}
	catch (RpcException exception)
	{
		outcome = $"{exception.StatusCode}: {exception.Status.Detail}";
	}
	outcomes[outcome] = outcomes.GetValueOrDefault(outcome) + 1;
}
foreach (var (outcome, count) in outcomes) Console.WriteLine($"{count} of {runs}: {outcome}");

static int FreePort()
{
	using var listener = new TcpListener(IPAddress.Loopback, 0);
	listener.Start();
	return ((IPEndPoint)listener.LocalEndpoint).Port;
}

Run: dotnet build -c Release, then dotnet bin/Release/net10.0/Repro.dll 200 and dotnet bin/Release/net10.0/Repro.dll 200 load.

Observed on Linux (.NET 10.0.14, 2 processors): 17 of 200 runs without load and 54 of 200 under thread-pool load end with the exception; on Windows 11 (.NET runtime 10.0.13, SDK 10.0.402): 2 of 200 without load and 8 of 200 under load.

Activity

  1. dotnet-policy-service commented on Nov 2, 2026

    @dotnet-policy-service
    Contributor

    Tagging subscribers to this area: @karelz, @dotnet/ncl
    See info in area-owners.md if you want to be subscribed.

  2. ManickaP commented on Nov 3, 2026

    @ManickaP
    Member

    This needs less info. This is obviously LLM generated report. We need human written summary, description of real-world scenario where you encountered it. Or, if this is based only on LLM analysis of source code, we need to know that as well to appropriately triage it.

  3. wolfgangcolsman commented on Nov 3, 2026

    @wolfgangcolsman
    Author

    Fair enough. Yes, I used an AI agent to write the report, and sorry about the length.

    How we ran into it: we have a service that hosts a gRPC endpoint on Kestrel, and other services connect to it with Grpc.Net.Client over long-lived duplex streams. When we shut that endpoint down, we end each call with Unavailable and a message in the trailers, then stop Kestrel. One of our CI tests checks that the client sees that status. Now and then the client gets Internal with "The HTTP/2 server closed the connection ... NO_ERROR" instead, and that happens more often when the machine is busy.

    The repro in the issue uses only Kestrel and Grpc.Net.Client. The explanation of why it happens inside SocketsHttpHandler comes from the agent reading the source. I haven't confirmed it with a debugger or a packet capture.

    We worked around it by keeping the connection open after the shutdown until the client closes it, so it's not blocking us. I also filed the gRPC side as grpc/grpc-dotnet#2786.

  4. ManickaP commented on Nov 3, 2026

    @ManickaP
    Member

    I'm pretty positive that this is just a dupe of #135063.
    Feel free to share your findings there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions