Enable Helix queue stats logging for Arcade unit tests by chcosta · Pull Request #17018 · dotnet/arcade

chcosta · 2026-06-17T16:35:14Z

Enables the new opt-in Helix queue health summary (added in #16922) for Arcade's own tests/UnitTests.proj.

Adds <EnableShowHelixQueueStats>true</EnableShowHelixQueueStats> so that every Helix job submitted by the Arcade unit-test pipeline logs the queue health summary at submit time. This dogfoods the feature and gives FRs immediate signal on Helix queue conditions when Arcade's own CI submits work.

Depends on #16922 having shipped through Arcade.

Copilot

Pull request overview

Enables Helix queue health summary logging for Arcade’s own Helix-submitted unit test jobs by opting in via MSBuild property, so Arcade CI “dogfoods” the feature and emits queue condition signal at job submission time.

Changes:

Set EnableShowHelixQueueStats to true in tests/UnitTests.proj so Helix submissions from Arcade unit tests log the queue health summary.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Default CI verbosity is Minimal, which filters out MessageImportance.Normal messages. When EnableShowHelixQueueStats is opted in, route SendAsync log output (including the queue-health summary) at High importance so it survives the default verbosity.

chcosta · 2026-06-17T23:09:55Z

/azp run

azure-pipelines · 2026-06-17T23:10:07Z

Azure Pipelines successfully started running 1 pipeline(s).

chcosta · 2026-06-19T20:53:02Z

/azp run

azure-pipelines · 2026-06-19T20:53:14Z

Azure Pipelines successfully started running 1 pipeline(s).

chcosta · 2026-06-22T16:44:52Z

/azp run

azure-pipelines · 2026-06-22T16:45:06Z

Azure Pipelines successfully started running 1 pipeline(s).

Add WithAzureDevOpsOrganization() and WithAzureDevOpsProject() fluent builder methods to JobDefinition to allow callers (especially public pipelines) to specify which Azure DevOps organization/project is requesting queue stats. These parameters are passed through JobCreationRequest to Helix API. Fixes: Arcade PR for enabling queue stats support with org/project scoping

Queue names are globally unique across all organizations and projects, so the Helix API does not need org/project context to return stats. These parameters only serve as optional audit context on the public Helix Observer endpoint for external callers to declare their scope. Reverts the org/project fluent builder methods since: 1. They are unused in the codebase 2. Queue stats are queried and returned by queue name globally 3. Org/project scoping is only relevant on the public HTTP endpoint

chcosta · 2026-06-23T05:13:55Z

/azp run

azure-pipelines · 2026-06-23T05:14:09Z

Azure Pipelines successfully started running 1 pipeline(s).

chcosta · 2026-06-23T22:20:32Z

/azp run

azure-pipelines · 2026-06-23T22:20:45Z

Azure Pipelines successfully started running 1 pipeline(s).

chcosta · 2026-06-24T16:47:32Z

/azp run

azure-pipelines · 2026-06-24T16:47:45Z

Azure Pipelines successfully started running 1 pipeline(s).

…sta/enable-helix-queue-stats

chcosta · 2026-06-24T18:24:24Z

The service issues seem to be resolved now and this is ready for review. queue wait time estimates / stats are being shown in the Test SubmitHelixTests steps.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

chcosta · 2026-06-30T19:23:58Z

/azp run

azure-pipelines · 2026-06-30T19:24:11Z

Azure Pipelines successfully started running 1 pipeline(s).

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

…Request Queue stats are queried globally by queue name, so these request fields are not needed. Removing the leftover model properties that were added intentionally but ultimately unused. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Instead of elevating every SendAsync progress message to High importance (which floods the build at 'Minimal' verbosity) or fragile string matching, add a SendAsync overload with a separate queueStatsLog callback. The SDK routes routine progress at Normal importance and the opt-in queue health summary at High importance, so only the queue-stats lines survive the default 'Minimal' verbosity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

- Add default interface implementation for the new SendAsync(log, queueStatsLog) overload so external IJobDefinition implementers keep compiling (delegates to the single-callback overload and ignores queueStatsLog; documented on the interface). - Extract SendHelixJob.CreateSubmissionLoggers, a pure factory that builds the routine and queue-stats log callbacks, so the enabled/disabled -> High/Normal routing is behaviorally unit-testable; expose Sdk internals to Sdk.Tests. - Add tests: DIM delegation via a concrete legacy IJobDefinition stub (JobSender.Tests, asserts log + cancellation token are forwarded) and queue-stats importance routing for both flag states (Sdk.Tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

chcosta · 2026-07-03T15:57:00Z

Ok, this is ready for review, @mmitche @premun

chcosta · 2026-07-06T17:11:36Z

Here's an example of the change to pipeline behavior

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1492794&view=logs&j=21143b0a-ace1-5058-215e-b11404388d17&t=68957cf3-3aa0-5af2-df5a-5bb40d4d0ecf&l=564

  note : Helix queue health reporting is a preview feature; data and format may change.
  Helix queue 'ubuntu.2204.amd64.open' health:
    Estimated wait : 0s   (queue depth: 9, avg run: 3m 55s)
    Snapshot taken : 2026-07-03 06:15 Coordinated Universal Time
    Questions about Helix queue health? Reach the dnceng First Responders channel: https://teams.microsoft.com/l/channel/19%3Aafba3d1545dd45d7b79f34c1821f6055%40thread.skype/First%20Responders?groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47
  Sent Helix Job 'Arcade Job Name'; see work items at https://helix.dot.net/api/jobs/2d9a30f2-3a04-4b60-9fd2-ebdec71b4b5a/workitems?api-version=2019-06-17
  Sending Job to Windows.11.Amd64.Client.Open...
  note : Helix queue health reporting is a preview feature; data and format may change.
  Helix queue 'windows.11.amd64.client.open' health:
    Estimated wait : 0s   (queue depth: 6, avg run: 11m 40s)
    Snapshot taken : 2026-07-03 06:15 Coordinated Universal Time
    Questions about Helix queue health? Reach the dnceng First Responders channel: https://teams.microsoft.com/l/channel/19%3Aafba3d1545dd45d7b79f34c1821f6055%40thread.skype/First%20Responders?groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47
  Sent Helix Job 'Arcade Job Name'; see work items at https://helix.dot.net/api/jobs/2b291e69-2b31-4ea3-a16d-f863c22a41a7/workitems?api-version=2019-06-17
  Sending Job to (Debian.13.Amd64.Open)ubuntu.2204.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-13-helix-amd64...
  note : Helix queue health reporting is a preview feature; data and format may change.
  Helix queue 'ubuntu.2204.amd64.open' health:
    Estimated wait : 0s   (queue depth: 9, avg run: 3m 55s)
    Snapshot taken : 2026-07-03 06:15 Coordinated Universal Time
    Questions about Helix queue health? Reach the dnceng First Responders channel: https://teams.microsoft.com/l/channel/19%3Aafba3d1545dd45d7b79f34c1821f6055%40thread.skype/First%20Responders?groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47
  Sent Helix Job 'Arcade Job Name'; see work items at https://helix.dot.net/api/jobs/3ffb9616-5b40-4564-9ea8-caa7bf34da1f/workitems?api-version=2019-06-17
  Sending Job to OSX.26.Arm64.Open...
  note : Helix queue health reporting is a preview feature; data and format may change.
  Helix queue 'osx.26.arm64.open' health:
    Estimated wait : 0s   (queue depth: 37, avg run: 47s)
    Snapshot taken : 2026-07-03 06:15 Coordinated Universal Time
    Questions about Helix queue health? Reach the dnceng First Responders channel: https://teams.microsoft.com/l/channel/19%3Aafba3d1545dd45d7b79f34c1821f6055%40thread.skype/First%20Responders?groupId=4d73664c-9f2f-450d-82a5-c2f02756606d&tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47
  Sent Helix Job 'Arcade Job Name'; see work items at https://helix.dot.net/api/jobs/c61763f6-6d79-43ea-9687-cc09f62ae275/workitems?api-version=2019-06-17

The estimated wait time may or may not be accurate. This PR is about enabling the functionality. i have separate efforts to track and ensure accuracy of the wait time estimation.

missymessa · 2026-07-07T16:07:19Z

Are the tests in here enough? Did you want to add any to Arcade Validation?

Enable Helix queue stats logging for Arcade unit tests

2a0c522

Copilot AI review requested due to automatic review settings June 17, 2026 16:35

Copilot started reviewing on behalf of chcosta June 17, 2026 16:35 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

mmitche reviewed Jun 22, 2026

View reviewed changes

Comment thread src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs Outdated

chcosta added 2 commits June 22, 2026 18:22

Merge branch 'main' of https://github.com/dotnet/arcade into dev/chco…

e308631

…sta/enable-helix-queue-stats

Copilot AI review requested due to automatic review settings June 24, 2026 18:23

Copilot started reviewing on behalf of chcosta June 24, 2026 18:24 View session

chcosta marked this pull request as ready for review June 24, 2026 18:24

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Comment thread src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Models/JobCreationRequest.cs Outdated

Comment thread src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs Outdated

Potential fix for pull request finding

b5fb6bb

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings July 2, 2026 20:21

Copilot started reviewing on behalf of chcosta July 2, 2026 20:21 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs Outdated

Comment thread src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Models/JobCreationRequest.cs Outdated

chcosta and others added 3 commits July 2, 2026 13:29

Merge remote-tracking branch 'origin/main' into pr-fork-work

3aab9d8

Copilot AI review requested due to automatic review settings July 2, 2026 20:40

Copilot started reviewing on behalf of chcosta July 2, 2026 20:40 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

Comment thread src/Microsoft.DotNet.Helix/JobSender/IJobDefinition.cs Outdated

Comment thread src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs Outdated

chcosta requested a review from premun July 3, 2026 15:57

chcosta requested review from epananth and missymessa July 6, 2026 17:11

missymessa approved these changes Jul 7, 2026

View reviewed changes

chcosta merged commit 1690e7d into dotnet:main Jul 7, 2026
15 checks passed

chcosta deleted the dev/chcosta/enable-helix-queue-stats branch July 7, 2026 20:28

dotnet-maestro Bot mentioned this pull request Jul 11, 2026

[main] Source code updates from dotnet/arcade dotnet/dotnet#7667

Merged

dotnet-milestone-bot Bot added this to the 11.0-preview7 milestone Jul 13, 2026

Uh oh!

Conversation

chcosta commented Jun 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

chcosta commented Jun 17, 2026

Uh oh!

azure-pipelines Bot commented Jun 17, 2026

Uh oh!

chcosta commented Jun 19, 2026

Uh oh!

azure-pipelines Bot commented Jun 19, 2026

Uh oh!

chcosta commented Jun 22, 2026

Uh oh!

azure-pipelines Bot commented Jun 22, 2026

Uh oh!

Uh oh!

chcosta commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jun 23, 2026

Uh oh!

chcosta commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jun 23, 2026

Uh oh!

chcosta commented Jun 24, 2026

Uh oh!

azure-pipelines Bot commented Jun 24, 2026

Uh oh!

chcosta commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

chcosta commented Jun 30, 2026

Uh oh!

azure-pipelines Bot commented Jun 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

chcosta commented Jul 3, 2026

Uh oh!

chcosta commented Jul 6, 2026

Uh oh!

missymessa commented Jul 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants