FEAT Add CyberSecEval MITRE dataset loader

[Varun Poojari (Varunpoojari)]

Description

Adds _CyberSecEvalMitreDataset, a remote loader for the English MITRE benchmark of Meta's CyberSecEval (Purple Llama). Closes #3003 and contributes the dataset-loading portion of the broader CyberSecEval parity tracked in #511. This is additive relative to v1.1.0; existing exports are retained.

• Fetches 1,000 prompts, 100 for each of ten ATT&CK categories, from PurpleLlama commit 172c1074069eb88ec834124272c1b1c4f8893445. The benchmarks are MIT licensed.
• Emits the upstream mutated_prompt verbatim as a literal SeedPrompt, including its wrapper. Preserves base_prompt, mitre_category, ttp_id, and ttp_name in metadata, plus the harm-mapping version. Each seed's source identifies the pinned dataset file.
• Exposes the ten-category CyberSecEvalMitreCategory enum filter. Empty filters, empty results, malformed categories/base prompts, and non-string prompt values raise explicit errors. Empty/None prompt values retain the warning-and-skip behavior.
• Keeps synchronous fetching/cache I/O off the event loop through asyncio.to_thread.
• Separates ATT&CK tactics from standardized harms. All rows receive COORDINATION_HARM; C2, Evasion, Execution, Lateral Movement, Persistence, and Privilege Escalation additionally receive MALWARE. Recon and Discovery are not automatically classified as malware.
• Adds content-reviewed, exact-prompt privacy annotations for Collection/Exfil rather than keyword heuristics or trusting noisy technique names. At the pinned source, 42 distinct curated prompts add malware/privacy labels: 39 carry PPI, 13 carry PROPRIETARY_INFO, with ten overlapping. The complete dataset has 642 MALWARE rows. Modified/custom prompt text does not inherit an exact-match privacy annotation.
• Declares text modality, large size, safety/cybersecurity tags, and all four harm categories for discovery. The dataset is not tagged default because it is cybersecurity-specific.

These are benchmark capability labels, not ground-truth judgments that every dual-use response is harmful. Upstream tactic/technique metadata is retained even where noisy, including one TA0043 tactic ID. Expansion/judging, false-refusal calibration, multilingual prompts, and a full CyberSecEval execution scenario remain out of scope. Literal prompts need separately authored objectives when adapted to standard attack scenarios; some upstream base prompts contain generation artifacts such as #ERROR!.

Tests and Documentation

Added 48 loader tests covering literal values and metadata, all ten tactic mappings, filters, validation, origin, nonblocking fetching, exact-match privacy annotations, changed-text rejection of those annotations, real SQLite persistence/querying, and execution of the documented balanced-selection cell with mocked external I/O.

Updated the paired loading-datasets notebook/Python file, citation, and bibliography. The new example loads into memory, independently samples exactly three prompts per tactic (30 distinct prompts), and explains per-category AtomicAttack.display_group reporting. Existing notebook outputs are preserved.

Validation performed on the implementation:

• uv run --frozen --no-sync python -m pytest tests\unit\datasets\test_cyberseceval_mitre_dataset.py -q: 48 passed.
• uv run --frozen --no-sync python -m pytest tests\unit\datasets\test_cyberseceval_mitre_dataset.py tests\unit\datasets\test_seed_dataset_provider.py tests\unit\datasets\test_seed_dataset_metadata.py tests\unit\datasets\test_remote_dataset_origin.py tests\unit\models\test_harm_category.py -q: 357 passed, including after main integration.
• uv run --frozen --no-sync pre-commit run --files pyrit\datasets\seed_datasets\remote\cyberseceval_mitre_dataset.py tests\unit\datasets\test_cyberseceval_mitre_dataset.py doc\code\datasets\1_loading_datasets.py doc\code\datasets\1_loading_datasets.ipynb: passed, including Ruff, documentation hooks, and the full production type check.
• uv run --frozen --no-sync ty check pyrit\datasets\seed_datasets\remote\cyberseceval_mitre_dataset.py tests\unit\datasets\test_cyberseceval_mitre_dataset.py: passed.
• uv run --frozen --no-sync python -m build_scripts.validate_docs: passed, 166 references and no orphaned documentation files.
• uv run --frozen --no-sync jupytext --update --to ipynb doc\code\datasets\1_loading_datasets.py: passed. The new balanced-selection cell was separately executed against the real pinned source in the worktree's uv environment with isolated memory/cache, and its retained output was verified. The entire older notebook was not re-executed.
• Source/memory probes via uv run --frozen --no-sync python -: passed. All 42 curated fingerprints resolve to real source rows; all 1,000 literal values remain unchanged; stored/queryable counts are COORDINATION_HARM=1000, MALWARE=642, PPI=39, PROPRIETARY_INFO=13; all ten sampled groups contain exactly three distinct prompts. Paired cell sources match and original outputs are unchanged.
• The existing real-source tests\end_to_end\test_all_datasets.py -k CyberSecEvalMitre -q case, invoked with pytest.main via uv run --frozen --no-sync python - and an isolated cache: 1 passed, 143 deselected.

Latest main has been merged without rebasing or force-pushing. Required current-head CI is being checked before the normal protected squash merge.

[Varun Poojari (Varunpoojari)]

@microsoft-github-policy-service agree

[Varun Poojari (Varunpoojari)]

Thanks for the review, Roman Lutz (@romanlutz)! Both points are addressed:

• mutated_prompt values that are not strings now raise a ValueError naming the row. Empty strings and None still go through the existing warning-and-skip path. Added object, array and number cases.
• The fetch now runs through await asyncio.to_thread(self._fetch_from_url, ...). Added a concurrency test with a 300 ms fetch stub and a 10 ms heartbeat; it fails against the previous blocking call and passes now.

I also merged the latest main to resolve the conflicts in references.bib and bibliography.md.