`NegotiateClientCertificateAsync` throws on OpenSSL 1.1.1 when the client doesn't offer post-handshake auth

[caraioniurie47]

Description

On Linux with OpenSSL 1.1.1, SslStream.NegotiateClientCertificateAsync on a TLS 1.3 connection whose client didn't offer post-handshake authentication throws OpenSslCryptographicException (error:14268117:SSL routines:SSL_verify_client_post_handshake:extension not received). With OpenSSL 3 the same call returns and RemoteCertificate stays null, as #128942 intended.

In Interop.OpenSsl.SslRenegotiate, #128942 recognizes the case by masking the exception's HResult with 0X7FFFFF, at Interop.OpenSsl.cs:696-701:

SecurityStatusPalErrorCode palErrorCode = (ex?.HResult & 0X7FFFFF) switch
{
    279 /*SSL_R_EXTENSION_NOT_RECEIVED*/ or
    339 /*SSL_R_NO_RENEGOTIATION*/ => SecurityStatusPalErrorCode.NoRenegotiation,
    _ => SecurityStatusPalErrorCode.InternalError
};

The HResult is the packed OpenSSL error code, Interop.ERR.cs:111-115:

internal OpenSslCryptographicException(int errorCode, string message)
    : base(message)
{
    HResult = errorCode;
}

0X7FFFFF is OpenSSL 3's ERR_REASON_MASK: there the code is lib << 23 | reason, 0x0A000117 for this error. OpenSSL 1.1 packs lib << 24 | func << 12 | reason, so the same error is 0x14268117, which masks to 0x268117, and the call falls through to InternalError.

Reproduction Steps

A console app from dotnet new console -n repro (.NET 11 SDK), with this Program.cs:

using System;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Authentication;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

[assembly: System.Runtime.Versioning.SupportedOSPlatform("linux")]

Console.WriteLine($"OpenSSL 0x{SafeEvpPKeyHandle.OpenSslVersion:X}");

using RSA key = RSA.Create(2048);
CertificateRequest request = new("CN=localhost", key, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
using X509Certificate2 ephemeral = request.CreateSelfSigned(DateTimeOffset.UtcNow.AddDays(-1), DateTimeOffset.UtcNow.AddDays(1));
using X509Certificate2 serverCert = X509CertificateLoader.LoadPkcs12(ephemeral.Export(X509ContentType.Pfx), null);

using TcpListener listener = new(IPAddress.Loopback, 0);
listener.Start();
using TcpClient tcpClient = new();
Task connect = tcpClient.ConnectAsync(IPAddress.Loopback, ((IPEndPoint)listener.LocalEndpoint).Port);
using TcpClient tcpServer = await listener.AcceptTcpClientAsync();
await connect;

using SslStream client = new(tcpClient.GetStream());
using SslStream server = new(tcpServer.GetStream());

// The client sets no certificate, so it doesn't offer TLS 1.3 post-handshake authentication.
await Task.WhenAll(
    client.AuthenticateAsClientAsync(new SslClientAuthenticationOptions
    {
        TargetHost = "localhost",
        EnabledSslProtocols = SslProtocols.Tls13,
        RemoteCertificateValidationCallback = delegate { return true; },
    }),
    server.AuthenticateAsServerAsync(new SslServerAuthenticationOptions { ServerCertificate = serverCert }));
Console.WriteLine($"Negotiated {server.SslProtocol}");

// Exchange data so the TLS 1.3 handshake is complete on both sides.
byte[] buffer = new byte[1];
await client.WriteAsync(new byte[] { 1 });
await server.ReadExactlyAsync(buffer);
await server.WriteAsync(new byte[] { 2 });
await client.ReadExactlyAsync(buffer);

// The client must be reading for the server's request to be processed.
ValueTask<int> clientRead = client.ReadAsync(buffer);
try
{
    await server.NegotiateClientCertificateAsync();
    Console.WriteLine($"NegotiateClientCertificateAsync returned; RemoteCertificate is {(server.RemoteCertificate is null ? "null" : "set")}");
}
catch (Exception e)
{
    Console.WriteLine($"NegotiateClientCertificateAsync threw {e.GetType()}: {e.Message}");
}

await server.WriteAsync(new byte[] { 3 });
await clientRead;
Console.WriteLine("Stream still usable");

Run it where .NET loads OpenSSL 1.1.1. On Ubuntu 22.04 x64 (OpenSSL 3.0.2), Ubuntu 20.04's libssl1.1 unpacked beside the system library does it, from the repro directory:

curl -O http://archive.ubuntu.com/ubuntu/pool/main/o/openssl/libssl1.1_1.1.1f-1ubuntu2_amd64.deb
dpkg-deb -x libssl1.1_1.1.1f-1ubuntu2_amd64.deb ssl111
dotnet build -c Release
DOTNET_OPENSSL_VERSION_OVERRIDE=1.1 LD_LIBRARY_PATH=$PWD/ssl111/usr/lib/x86_64-linux-gnu dotnet bin/Release/net11.0/repro.dll

Expected behavior

The call returns, as it does with OpenSSL 3.0.2:

OpenSSL 0x30000020
Negotiated Tls13
NegotiateClientCertificateAsync returned; RemoteCertificate is null
Stream still usable

Actual behavior

With OpenSSL 1.1.1f:

OpenSSL 0x1010106F
Negotiated Tls13
NegotiateClientCertificateAsync threw Interop+Crypto+OpenSslCryptographicException: error:14268117:SSL routines:SSL_verify_client_post_handshake:extension not received
Stream still usable

Regression?

No. On .NET 10.0.12 the same repro (retargeted to net10.0) throws Interop+OpenSsl+SslException: Operation failed with error - 0. with both OpenSSL 3.0.2 and 1.1.1f. #128942 made the call return on .NET 11, but its check recognizes only OpenSSL 3's error layout.

Known Workarounds

The stream stays usable after the exception, so a server can catch CryptographicException around the call and treat it as "no certificate", at the cost of treating any other OpenSSL failure of the call the same way.

Configuration

.NET 11.0.0-rc.1.26425.128, Ubuntu 22.04 x64 (WSL2), OpenSSL 1.1.1f loaded as above; OpenSSL 3.0.2 on the same machine is not affected. RHEL 8 is on .NET 11's supported OS list, and its derivative Rocky Linux 8 ships openssl-libs 1.1.1k.

Other information

SslStream_NegotiateClientCertificateAsync_Tls13PhaNotOffered, the test #128942 added, fails on OpenSSL 1.1.1f with the same exception.

TlsSession.RequestClientCertificate on a socket-bound session goes through the same check: in the same scenario it returns Complete with OpenSSL 3.0.2 and throws AuthenticationException with OpenSSL 1.1.1f, the exception above as its inner exception.

I have a fix that decodes the reason with the layout of the loaded OpenSSL, only for errors from the SSL library, with a unit test for both layouts and a test for the TlsSession case, and I'll send it as a PR. Could this be assigned to me?

Searching issues and PRs for Tls13PhaNotOffered, 14268117 and "extension not received" found no existing report.

Note

AI-generated, written at my direction and reviewed by me before posting. The repro was compiled and run as written on .NET 11.0.0-rc.1.26425.128 and, retargeted to net10.0, on .NET 10.0.12, on Ubuntu 22.04 x64 with OpenSSL 3.0.2 and 1.1.1f. The test results come from a local Release build of main at 99f3a67, which is also where the excerpts are from. The error-code layouts come from the OpenSSL 1.1.1t and 3.0.2 headers.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.