This is an internal testing/infrastructure contribution, not a customer-facing product incident. The core version is deliberately pinned for reproducibility rather than upgraded implicitly.
Output from azd version
Run azd version and copy and paste the output here:
The installed-package validation uses azd 1.33.0, with its release archive checksum and executable digest recorded. The public package baseline is evaluations 1.0.41-beta and dataset 1.0.0-beta.29.
Describe the bug
Description of issue you're seeing...
The existing candidate proof validates a manually frozen candidate through 160 installed CLI checks on each tested OS and two source-race jobs. We also need a separate, shared scenario-CI entry point that resolves the public feed's Latest release once, freezes its identity for parallel jobs, exercises installed offline behavior, and produces auditable sanitized evidence in GitHub Actions and a reusable Azure DevOps pipeline definition.
To Reproduce
Steps to reproduce the behavior...
- Inspect
.github/workflows/eval-candidate-proof.yml and eng/scripts/eval-candidate-proof/.
- Observe that this fixed-candidate release gate is not a Latest-resolution/shared-provider scenario runner.
- Keep that release gate unchanged while adding the independent internal validation surface.
Expected behavior
A clear and concise description of what you expected to happen.
- Reuse the existing 160 installed-CLI checks without changing their contract or critical release workflow.
- Resolve Latest once; record immutable release/source/version/registry/archive pins and verify actual installed executable bytes and runtime versions.
- Add bounded offline configuration-isolation and cancellation-argument refusal scenarios, with exact JSON/exit/state assertions. Do not describe these as interactive cancellation proof.
- Supply a shared runner, unit tests, an actual verified GitHub Actions offline run with downloadable evidence, and a locally validated Azure DevOps YAML entry point.
- Report Azure DevOps execution as BLOCKED until an authorized existing organization/project/repository connection/pipeline/capacity tuple is supplied. This initial contribution does not create those resources.
- Fail closed on requests for service-backed execution while identity/resource/budget prerequisites are absent. No credentials export, new IAM, runner registration, paid generation, or deployment.
- Keep release acceptance documentation and contributor review-readiness guidance aligned with observed evidence.
Environment
Information on your environment:
* Language name and version: Python 3.12 standard-library runner; actual azd/extension release binaries.
* IDE and version: N/A. GitHub-hosted Ubuntu 24.04 and Windows 2025 runners; Azure DevOps execution target not authorized/configured.
Additional context
Add any other context about the problem here.
Deduplication found #10110, which requests a customer-copyable, authenticated live Azure quality-gate sample. This internal offline-testing contribution is deliberately narrower and must not close or claim to fulfill #10110, revive its sample PR, or count source/HTTP fixtures as live service results. Authenticated Azure DevOps/service execution remains a separately authorized follow-up, not a fabricated success in this issue's initial scope.
This is an internal testing/infrastructure contribution, not a customer-facing product incident. The core version is deliberately pinned for reproducibility rather than upgraded implicitly.
Output from
azd versionRun
azd versionand copy and paste the output here:The installed-package validation uses
azd1.33.0, with its release archive checksum and executable digest recorded. The public package baseline is evaluations 1.0.41-beta and dataset 1.0.0-beta.29.Describe the bug
Description of issue you're seeing...
The existing candidate proof validates a manually frozen candidate through 160 installed CLI checks on each tested OS and two source-race jobs. We also need a separate, shared scenario-CI entry point that resolves the public feed's Latest release once, freezes its identity for parallel jobs, exercises installed offline behavior, and produces auditable sanitized evidence in GitHub Actions and a reusable Azure DevOps pipeline definition.
To Reproduce
Steps to reproduce the behavior...
.github/workflows/eval-candidate-proof.ymlandeng/scripts/eval-candidate-proof/.Expected behavior
A clear and concise description of what you expected to happen.
Environment
Information on your environment:
* Language name and version: Python 3.12 standard-library runner; actual azd/extension release binaries.
* IDE and version: N/A. GitHub-hosted Ubuntu 24.04 and Windows 2025 runners; Azure DevOps execution target not authorized/configured.
Additional context
Add any other context about the problem here.
Deduplication found #10110, which requests a customer-copyable, authenticated live Azure quality-gate sample. This internal offline-testing contribution is deliberately narrower and must not close or claim to fulfill #10110, revive its sample PR, or count source/HTTP fixtures as live service results. Authenticated Azure DevOps/service execution remains a separately authorized follow-up, not a fabricated success in this issue's initial scope.