Skip to content

FEAT: Run a scenario over exactly selected dataset examples #2760

Open

Description

@romanlutz

Is your feature request related to a problem? Please describe.

The exact-selection contract needs a small ordinary scenario consumer. Other defaults could sample away selections, skip incompatible work, run unintended techniques or duplicate prompt sending as technique and baseline.

Part of #2744. Depends on #2757, including the completed scorer API. Keep not ready yet until promoted.

Describe the solution you'd like

Implement the smallest registered scenario/configuration adapter over resolved selections. Reuse Scenario, explicit DatasetAttackConfiguration groups, factories/builders, shared scorer policy, normal progress/history and saved-run plans. technique.execute deliberately handles one free-text objective without dataset inputs; reuse patterns, not repeated REST launches or an unrelated expansion of its contract.

Untouched settings produce exactly one prompt-sending execution per valid selected example: no additional converters, default cap, unintended technique set or duplicate baseline. Choose baseline/default behavior deliberately without changing other scenarios or adding a duplicate global factory as a workaround.

Preserve complete materialized conversations, per-turn media, roles/order/groups and provenance. Retain source objectives and require missing ones explicitly. Explain unsupported/ambiguous shapes and unresolved templates/configurations rather than flattening or generating them.

Reuse configured source-pinned scorers and shared default/expectation/condition validation. Validate the selected population and prerequisite targets before launch. True/false family alone is insufficient; raw refusal=true is not attack success. Invalid explicit choices must not silently fall back; no unscored mode.

Fail clearly for incompatible requested work rather than silently skipping examples/techniques or adapting selected content away. Reuse existing incompatible-technique policy/requirement checks, not a second service/DTO compatibility engine.

Acceptance criteria:

  • More than five valid selections still yield N default prompt-sending executions, not five, N plus a baseline or another default technique set.
  • Conversations/media, objectives/conditions and provenance survive execution and supported resume without reselecting/regenerating.
  • Invalid groups, missing objectives, incompatible targets/techniques and scorer prerequisites produce actionable pre-execution errors; nothing silently disappears.
  • Shared defaults, explicit non-default choices, stale references, composites/inverters and refusal-result meaning have focused coverage without duplicate policy.
  • Preview/estimate/launch agree; existing scenarios retain their defaults/named-source behavior.
  • Scenario/service tests use local fixtures and fake targets/scorers. Expose seams for existing Scanner UI, not a new runner.

Describe alternatives you've considered, if relevant

No GUI batch executor, generated scenario classes, global scorer redesign, provider loading, arbitrary config interpreter, memory migration or per-selection dataset. Optional additive converter UI #2758 is not required for the default consumer.

Additional context

Use current framework/scenario contracts, deliberate DatasetAttackConfiguration(seed_groups=..., max_total="all"), and saved run-plan/resume behavior. Current core already supplies condition validation and incompatible-technique policy; extend only demonstrated consumer gaps in the shared owner.

#2759 owns the existing-form integration and real-backend journey. Open #3526 is related form work, not a backend-adapter prerequisite. Scope narrowed November 4, 2026.

Activity

  1. added
    not ready yetThis issue needs more definition or is blocked by a pending change.
    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUI
    on Sep 22, 2026
  2. changed the title [-]FEAT: Support custom scenarios over selected dataset examples[/-] [+]FEAT: Run a scenario over exactly selected dataset examples[/+] on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    GUIUmbrella label for all feedback submitted via the Co-PyRIT GUIfeature-requestnot ready yetThis issue needs more definition or is blocked by a pending change.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions