[android][clr] No peer certificates when executing System.Net.Http.Functional.Tests on Android emulator

[matouskozak]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1297669
Build error leg or test failing: System.Net.Http.Functional.Tests
Runtime changes: c1cb909...fbf6be5

                 02-17 19:11:51.890 10934 25477 I DOTNET  : 	[IGNORED] System.Net.Http.Functional.Tests.SocketsHttpHandler_DiagnosticsTest_Http11.SendAsync_OperationCanceledException_RecordsActivitiesWithCorrectErrorInfo
                 02-17 19:11:51.891 10934 25477 I DOTNET  : System.Net.Http.Functional.Tests.SocketsHttpHandler_DiagnosticsTest_Http11 1.4314913 ms
                 02-17 19:11:51.891 10934 25477 I DOTNET  : 
                 02-17 19:11:51.891 10934 25477 I DOTNET  : Test collection for System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async
                 02-17 19:11:51.893 10934 25477 D DOTNET  : GetJNIEnv: Registering JNI thread detach. env ptr 0x7dcd5de753c0. Key: -2147483632
                 02-17 19:11:53.125 10934 25477 D DOTNET  : DetachThreadFromJNI: Detaching thread from JNI
                 02-17 19:11:54.064 10934 10945 I unctional.Test: Background young concurrent copying GC freed 129004(5772KB) AllocSpace objects, 0(0B) LOS objects, 24% free, 14MB/18MB, paused 381.045ms total 425.702ms
                 02-17 19:11:54.900 10934 25377 D DOTNET  : GetJNIEnv: Registering JNI thread detach. env ptr 0x7dcd5de75500. Key: -2147483632
                 02-17 19:11:54.900 10934 25377 W System.err: javax.net.ssl.SSLPeerUnverifiedException: No peer certificates
                 02-17 19:11:54.901 10934 25377 W System.err: 	at com.android.org.conscrypt.ActiveSession.checkPeerCertificatesPresent(ActiveSession.java:314)
                 02-17 19:11:54.901 10934 25377 W System.err: 	at com.android.org.conscrypt.ActiveSession.getPeerCertificates(ActiveSession.java:180)
                 02-17 19:11:54.901 10934 25377 W System.err: 	at com.android.org.conscrypt.ExternalSession.getPeerCertificates(ExternalSession.java:106)
                 02-17 19:11:54.901 10934 25377 W System.err: 	at com.android.org.conscrypt.Java7ExtendedSSLSession.getPeerCertificates(Java7ExtendedSSLSession.java:129)
                 02-17 19:11:54.901 10934 25377 W System.err: 	at com.android.org.conscrypt.Java7ExtendedSSLSession.getPeerCertificates(Java7ExtendedSSLSession.java:31)
                 02-17 19:11:55.010 10934 25485 I DOTNET  : 	[PASS] System.Net.Http.Functional.Tests.SocksProxyTest_Http1_Async.TestLoopbackAsync

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": [
      "System.Net.Http.Functional.Tests",
      "System.err: javax.net.ssl.SSLPeerUnverifiedException: No peer certificates"
  ],
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1297669
Error message validated: [System.Net.Http.Functional.Tests System.err: javax.net.ssl.SSLPeerUnverifiedException: No peer certificates]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/17/2026 8:00:42 PM UTC

Report

Build	Definition	Test	Pull Request
1309353	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124833
1309331	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124850
1309271	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124854
1309257	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#123145
1309147	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124766
1309150	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124851
1309133	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124603
1309118	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124791
1309105	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124650
1308132	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124765
1309027	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#117956
1308999	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124753
1309007	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1308953	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124849
1308845	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124842
1308941	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124650
1308404	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124791
1308935	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124846
1308905	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124203
1308876	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.LargeUriAndHeaders_Works	#124760
1308825	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124731
1308801	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1308820	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124843
1308792	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124833
1308779	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124701
1308718	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124778
1308614	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124838
1308592	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124837
1308167	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124810
1308204	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124819
1308580	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124836
1308519	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124834
1308505	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124833
1308461	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124817
1308071	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124738
1308498	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#121906
1308476	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124832
1308412	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124830
1308467	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124831
1308424	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124760
1308450	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#123830
1308435	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124818
1308290	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124736
1308179	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124813
1308300	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124822
1308213	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124821
1308220	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124820
1308080	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.LargeUriAndHeaders_Works	#124782
1308149	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#120275
1308138	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124805
1308115	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1307983	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124203
1308022	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124300
1307890	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#122649
1308009	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124809
1307919	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124804
1297561	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124519
1307873	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124806
1307813	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124801
1307837	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124650
1306322	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124684
1307754	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124798
1307676	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124753
1307694	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124516
1307684	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124794
1307597	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124796
1307590	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#123323
1307570	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124764
1307543	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124649
1307524	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124566
1307536	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124791
1306121	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124567
1307513	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124792
1307261	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124589
1307507	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124590
1307493	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124273
1307470	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124593
1307463	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#123251
1307360	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124789
1307356	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.LargeUriAndHeaders_Works	#124788
1307341	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124650
1307351	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124646
1307312	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#123138
1307170	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#122526
1307166	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1307150	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124397
1307039	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#123939
1307035	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124586
1307000	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124738
1306959	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1306766	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124731
1306821	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#123614
1306872	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124203
1306890	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124778
1306655	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#120275
1306756	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124685
1306682	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.LargeUriAndHeaders_Works	#124773
1306737	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#122533
1306689	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124772
1306554	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#124763
Displaying 100 of 493 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
72	428	493

Report

Build	Definition	Test	Pull Request
1418874	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1418163	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1418156	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1414294	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1412581	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1409045	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1404155	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1402968	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1394388	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1391817	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1390570	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1390399	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1389302	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1384035	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1382941	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1382164	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1381576	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1381361	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1380981	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	4	19

[EgorBo]

@dotnet/ncl Can someone take a look at this? it has 365 hits per week on CI, all JIT PRs are always red because of it.

[ManickaP]

This is android specific AFAICS, isn't it?
cc @simonrozsival

[simonrozsival]

@ManickaP yes

[wfurt]

we may need to disable the offending tests if real fix drags too long.....

[vitek-karas]

Investigation Summary: Android Emulator CI Failure

Root Cause

The failures are caused by an Android Emulator version upgrade (36.2.x → 36.4.9) picked up during Helix VMSS re-imaging on Feb 14-15.

Timeline

Date	Event
Nov 12, 2025	Base Helix image built (Helix-Ubuntu-2204-2025-11-12) — emulator ≤36.2.12
Feb 10, 2026	Google releases Android Emulator 36.4.9 Stable with major networking changes (netsimd replaces QEMU pipe networking, libslirp stack updated, Lavapipe as default software renderer)
Feb 11-14	dotnet-helix-machines PRs (including PR 57760 adding linux-ubuntu-disable-unattended-upgrades to Android queues) trigger VMSS re-imaging
Feb 14, 20:01 UTC	Last green rolling build (1294907) — likely ran on a VM not yet re-imaged
Feb 15, 08:01 UTC	First red rolling build (1295230) — re-imaged VM with emulator 36.4.9

Evidence

• Not a code change: The 6 commits between the last green and first red builds touch zero SSL/HTTP/Android files.
• Systemic: Verified across 5 failing builds on 5 different Helix machines — all show identical symptoms (timeout at 1800s, exit code 70, 0 test results).
• QEMU pipe errors: Failing builds show 6-11 Failed to open QEMU pipe 'qemud:network': Invalid argument errors vs ~1 in green builds. This is consistent with emulator 36.2.6+ changing the WiFi packets handler to go through netsimd instead of QEMU pipes — API 29 system images still try the old qemud:network pipe, which no longer works properly.
• Unpinned emulator version: The Helix onboarding script (linux-ubuntu-android-emulator.sh) uses sdkmanager ... "emulator" which always installs the latest version — so re-imaging picked up 36.4.9.

Recommended Fix

Pin the Android emulator version in the dotnet-helix-machines repo's linux-ubuntu-android-emulator.sh to 36.2.12 (last known-good stable from Oct 13, 2025), or to 36.3.10 (Dec 4, 2025 stable). This prevents future uncontrolled emulator upgrades from breaking CI.

[wfurt]

Do you know what package name we need to pin @vitek-karas ? Also, there was debate with @simonrozsival and @akoeplinger that we may switch to docker images. That would give you full control over all aspects. But I'm not sure if we are ready....but it would be nice.

[vitek-karas]

I'm not convinced that pinning the version is the right solution - we already have way too many things which need "regular" manual updates - it adds too much cost to the maintenance. Historically I'm not aware of Android emulator updates causing problems - so if this is a one-off - I would personally prefer to keep it auto-updating and dealing with rare occasional breaks when they happen.
Similar for docker images - yes - it would give us full control - but it comes with the same maintenance cost.

[wfurt]

Agreed. While pinning may not be best option I think it would be ok to do it for now to get the CI green. We should keep looking for better strategy. But it is not clear to me what should be done right now and who is expected to do it. Also is this something that can be fixed via emulator config? I don't think @dotnet/ncl has enough insight to the Android details.