Skip to content

[Issue] Add internal installed-package evaluation scenario CI #10176

Description

  • Make sure you've installed the latest version using instructions

This is an internal testing/infrastructure contribution, not a customer-facing product incident. The core version is deliberately pinned for reproducibility rather than upgraded implicitly.

Output from azd version
Run azd version and copy and paste the output here:

The installed-package validation uses azd 1.33.0, with its release archive checksum and executable digest recorded. The public package baseline is evaluations 1.0.41-beta and dataset 1.0.0-beta.29.

Describe the bug
Description of issue you're seeing...

The existing candidate proof validates a manually frozen candidate through 160 installed CLI checks on each tested OS and two source-race jobs. We also need a separate, shared scenario-CI entry point that resolves the public feed's Latest release once, freezes its identity for parallel jobs, exercises installed offline behavior, and produces auditable sanitized evidence in GitHub Actions and a reusable Azure DevOps pipeline definition.

To Reproduce
Steps to reproduce the behavior...

  1. Inspect .github/workflows/eval-candidate-proof.yml and eng/scripts/eval-candidate-proof/.
  2. Observe that this fixed-candidate release gate is not a Latest-resolution/shared-provider scenario runner.
  3. Keep that release gate unchanged while adding the independent internal validation surface.

Expected behavior
A clear and concise description of what you expected to happen.

  • Reuse the existing 160 installed-CLI checks without changing their contract or critical release workflow.
  • Resolve Latest once; record immutable release/source/version/registry/archive pins and verify actual installed executable bytes and runtime versions.
  • Add bounded offline configuration-isolation and cancellation-argument refusal scenarios, with exact JSON/exit/state assertions. Do not describe these as interactive cancellation proof.
  • Supply a shared runner, unit tests, an actual verified GitHub Actions offline run with downloadable evidence, and a locally validated Azure DevOps YAML entry point.
  • Report Azure DevOps execution as BLOCKED until an authorized existing organization/project/repository connection/pipeline/capacity tuple is supplied. This initial contribution does not create those resources.
  • Fail closed on requests for service-backed execution while identity/resource/budget prerequisites are absent. No credentials export, new IAM, runner registration, paid generation, or deployment.
  • Keep release acceptance documentation and contributor review-readiness guidance aligned with observed evidence.

Environment
Information on your environment:
* Language name and version: Python 3.12 standard-library runner; actual azd/extension release binaries.
* IDE and version: N/A. GitHub-hosted Ubuntu 24.04 and Windows 2025 runners; Azure DevOps execution target not authorized/configured.

Additional context
Add any other context about the problem here.

Deduplication found #10110, which requests a customer-copyable, authenticated live Azure quality-gate sample. This internal offline-testing contribution is deliberately narrower and must not close or claim to fulfill #10110, revive its sample PR, or count source/HTTP fixtures as live service results. Authenticated Azure DevOps/service execution remains a separately authorized follow-up, not a fabricated success in this issue's initial scope.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/pipelineCI/CD pipeline config (GH Actions, AzDO)enhancementNew feature or improvementext-datasetazure.ai.dataset extensionext-evalsazure.ai.evaluations extensiontest automationTest infrastructure work

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions