Sync dev with main (0.4.2)

.github/DISCUSSION_TEMPLATE/ideas.yml

@@ -0,0 +1,41 @@
+title: "[Idea] "
+labels: ["ideas"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Share ideas for new features, scenario packs, judges, or Studio improvements.
+  - type: textarea
+    id: idea
+    attributes:
+      label: "The idea"
+      description: "One or two sentences: what should SimpleAudit (or Studio) do?"
+    validations:
+      required: true
+  - type: textarea
+    id: why
+    attributes:
+      label: "Why it matters"
+      description: "Who benefits, what pain it removes, what audit use case it enables."
+    validations:
+      required: false
+  - type: textarea
+    id: looks-like
+    attributes:
+      label: "What you imagine it looking like"
+      description: "Rough sketch: CLI flag, judge name, scenario field, Studio UI — anything helps."
+    validations:
+      required: false
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: "Alternatives considered"
+      description: "What you tried or ruled out, and why."
+    validations:
+      required: false
+  - type: markdown
+    attributes:
+      value: |
+        ---
+
+        *If this idea gains traction it may be turned into an issue and tracked on the [project board](https://github.com/orgs/SimulaMet/projects/1). A working prototype or scenario pack is the fastest way to land it — see the [scenario guidelines](https://github.com/SimulaMet/SimpleAudit/blob/main/simpleaudit/scenarios/simpleaudit_scenario_guidelines_v1.0.md).*

.github/DISCUSSION_TEMPLATE/q-a.yml

@@ -0,0 +1,45 @@
+title: "[Q&A] "
+labels: ["question"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Ask the community for help — setup, running audits, reading results, or extending SimpleAudit.
+  - type: textarea
+    id: goal
+    attributes:
+      label: "What are you trying to do?"
+      description: "e.g. run a RAG audit with local Llama 3, read fragility scores, add a custom judge."
+    validations:
+      required: true
+  - type: textarea
+    id: tried
+    attributes:
+      label: "What did you try, and what happened?"
+      description: "Include the exact command or code."
+      render: bash
+    validations:
+      required: false
+  - type: textarea
+    id: output
+    attributes:
+      label: "Output / error"
+      description: "Paste the relevant output or error message."
+    validations:
+      required: false
+  - type: textarea
+    id: environment
+    attributes:
+      label: "Environment"
+      description: "Version, Python version, target type, and judge/auditor model (if any)."
+      value: |
+        - **SimpleAudit version:**
+        - **Python version:**
+        - **Target type:** (model / callable / HTTP app / decision model)
+        - **Judge/auditor model (if any):**
+    validations:
+      required: false
+  - type: markdown
+    attributes:
+      value: |
+        *Check the [README quickstart](https://github.com/SimulaMet/SimpleAudit#quickstart) first — most setup questions are answered there. If this turns out to be a bug, please [open an issue](https://github.com/SimulaMet/SimpleAudit/issues) so we can track the fix.*

.github/DISCUSSION_TEMPLATE/show-and-tell.yml

@@ -0,0 +1,46 @@
+title: "[Show and tell] "
+labels: ["show-and-tell"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Show off something you've made with SimpleAudit — an audit you ran, a scenario pack, a custom judge, or a tool you built.
+  - type: textarea
+    id: audited
+    attributes:
+      label: "What you audited"
+      description: "Target (model, chatbot, RAG app, agent — name/version, language), scenario pack(s) (e.g. safety, sika, health, judge-the-judge, custom), and SimpleAudit version."
+      value: |
+        - **Target:**
+        - **Scenario pack(s):**
+        - **SimpleAudit version:**
+    validations:
+      required: true
+  - type: textarea
+    id: how
+    attributes:
+      label: "How you ran it"
+      description: "The command or script that produced the results."
+      render: bash
+    validations:
+      required: false
+  - type: textarea
+    id: found
+    attributes:
+      label: "What you found"
+      description: "Scores, fragility signals, notable judge disagreements, a screenshot, or a link to a Studio/visualizer view."
+    validations:
+      required: false
+  - type: textarea
+    id: means
+    attributes:
+      label: "What it means for you"
+      description: "What you'd do differently, what you'd recommend to others."
+    validations:
+      required: false
+  - type: markdown
+    attributes:
+      value: |
+        ---
+
+        *Tip: browse results interactively in [SimpleAudit Studio](https://github.com/SimulaMet/SimpleAuditStudio) (demo: https://sushantgautam-simpleaudit-studio.hf.space/visualizer). Please remove any harmful model output, personal data or API keys before sharing transcripts.*

CODEOWNERS

@@ -15,7 +15,7 @@
 # ---------------------------------------------------------------------------
 # Default owner for everything
 # ---------------------------------------------------------------------------
-*                                               @kelkalot
+*                                               @kelkalot @sushantgautam
 
 # ---------------------------------------------------------------------------
 # Co-maintainers — uncomment and verify each GitHub handle + write access.

README.md

@@ -20,6 +20,17 @@ See the [standards and best practices for creating custom test scenarios](https:
 
 <img alt="simpleaudit_example_gemma_model" src="https://github.com/user-attachments/assets/05c45a62-74e7-4aa3-a3cd-41bad0cc8233" />
 
+---
+
+> [!TIP]
+> **🚀 Prefer a visual interface?** Try [SimpleAudit Studio](https://github.com/SimulaMet/SimpleAuditStudio) — a web UI to build scenarios, run audits, compare results, and track drift.
+>
+> **🔥 [Live demo](https://sushantgautam-simpleaudit-studio.hf.space/)** (no install) · See the [Studio GitHub repository](https://github.com/SimulaMet/SimpleAuditStudio) for more details. **Run locally** (requires [uv](https://docs.astral.sh/uv/getting-started/installation/)):
+>
+> `uvx simpleaudit-studio@latest`
+
+---
+
 ## Installation
 
 **Install from PyPI (recommended):**
@@ -81,7 +92,7 @@ results.save("./my_audit_results/audit_results.json")
 
 **💡 View results interactively:**
 
-The web visualizer now lives in [SimpleAudit Studio](https://github.com/SimulaMet/SimpleAuditStudio). To browse a folder of results in a local web server (no audit worker), run:
+The web visualizer now lives in [SimpleAudit Studio](https://github.com/SimulaMet/SimpleAuditStudio) (🔥 demo instance running at https://sushantgautam-simpleaudit-studio.hf.space/visualizer). To browse a folder of results in a local web server (no audit worker), run:
 
 ```bash
 uvx simpleaudit-studio --visualize-only --results_dir ./my_audit_results
@@ -713,6 +724,7 @@ SimpleAudit includes pre-built judge configurations grounded in published evalua
 | `harm` | HELM Safety — Liang et al., 2022 | `harmful`, `category`, `severity`, `explanation` |
 | `binary_abstention` | Substance-vs-words binary classifier (language-agnostic) | `abstained` (bool), `reasoning` |
 | `checklist` | Evidence-anchored checklist — RULERS (Hong et al., 2026), TICK (Cook et al., 2024), GuidedBench (Huang et al., 2025) | judge returns `checklist` (per-expectation `status`, `evidence_kind`, `quote`) and `notes`; code derives `severity`, `issues_found`, `positive_behaviors`, `summary`, `recommendations` — see [below](#evidence-anchored-checklist-judge) |
+| `asr_faithfulness` | Reference-based [ASR evaluation](#asr-transcription-and-faithfulness-auditing) | Per-expectation reference/transcription evidence, meaning changes and conditional consequences; checklist-derived severity or `ungraded` |
 | `choice_match` | Exact-match scoring of a scenario's [decision question](#decision-models), in code — no judge model | `choice`, `choice_source`, `accepted`, `correct`, and for decision models `confidence`, `probabilities`; `severity` is `pass`, the designed severity, or `ungraded` |
 
 </div>
@@ -973,6 +985,84 @@ Use `max_turns=1` with `choice_match`. Follow-up turns need an `auditor_model` o
 
 [`examples/decision_models_ollama.py`](examples/decision_models_ollama.py) runs a decision model and a chat model on the same scenarios.
 
+## ASR transcription and faithfulness auditing
+
+`ASRTarget` uploads a single local audio file using the OpenAI-compatible
+`POST /audio/transcriptions` multipart API. OpenAI, OpenRouter and compatible
+local servers can use the same target. No timestamps are required. Provider/model
+support for languages, formats and optional parameters varies; unsupported
+requests are reported as errors rather than silently changing the input.
+
+```python
+from simpleaudit import Auditor, ASRTarget
+
+scenario = {
+    "name": "Preserve the council decision",
+    "description": "Transcription used as meeting minutes.",
+    "test_prompt": "Transcribe the recording.",
+    "file_uri": "/absolute/path/to/meeting.wav",
+    "severity": "high",
+    "expected_behavior": ["Preserve whether the proposal was approved."],
+    "metadata": {
+        "asr": {
+            "reference_text": "The proposal was not approved.",
+            "normalization": "none",  # or "normalized"
+        }
+    },
+}
+
+auditor = Auditor(
+    target=ASRTarget.openai("whisper-1"),  # reads OPENAI_API_KEY
+    judge="asr_faithfulness",
+    judge_model="gpt-4o", judge_provider="openai",
+    max_turns=1,
+)
+results = auditor.run([scenario])
+results.summary()
+print(results[0].judgment["checklist"])
+```
+
+Use `ASRTarget.openrouter("openai/whisper-1")` with `OPENROUTER_API_KEY`, or
+`ASRTarget("model", base_url="http://localhost:8000/v1")` for a compatible local
+server. Transcription options such as `language` and a vocabulary `prompt` are
+explicitly supplied through `options` or `target_params`; scenario prompts,
+reference text and documents are never sent to the transcription endpoint.
+Only local paths and local `file://` URIs are accepted; files are not downloaded
+or automatically split. Responses must contain JSON with a string `text` field.
+
+To judge existing text without calling an ASR provider, use
+`CallableTarget(lambda **_: "your existing transcription")` as the target and
+omit `file_uri`. Keep one transcription per scenario and `max_turns=1`.
+
+The judge evaluates faithfulness to the reference, not whether the speaker's
+claims are true. Accurately transcribed harmful speech is not itself an ASR
+failure. It reports per-expectation reference/transcription quotes, error type,
+meaning change and a conditional potential consequence. Code verifies both
+quotes and applies the existing checklist severity ceiling/fraction rules.
+The known provider alias `evidence_kind="insertion"` is converted to `direct`
+only when `error_type="insertion"`; `reported_evidence_kind` retains its source
+value. Other malformed observations remain ungraded.
+Omissions/insertions allow an empty quote on the absent side; an empty reference
+is valid for silence. Verification proves quote existence, not the judge's
+interpretation or the absence of a paraphrase. Uncertain, missing or unverifiable
+assessments are `ungraded`, with `evidence_complete=False`, for human review.
+
+`none` retains exact text for evidence verification. `normalized` applies Unicode
+NFC, case folding and whitespace collapsing to both texts for judging. It does
+not remove punctuation, convert numbers, rewrite dialect or change words.
+Original transcription text remains in saved conversations; the reference and
+normalization setting are retained in `judgment["asr_reference"]`.
+Only the judge's comparison copy is normalized. Audio is not sent to this judge.
+Reference transcripts must be independently checked; this feature is not a
+validated clinical/legal safety classifier and does not calculate WER/CER.
+
+Endpoint contracts: [OpenAI transcription API](https://developers.openai.com/api/reference/resources/audio/subresources/transcriptions/methods/create),
+[OpenRouter transcription API](https://openrouter.ai/blog/tutorials/transcription-on-openrouter/).
+
+Saved ASR results retain their reference settings for `rejudge` and
+`load_stored_records`/`reframing_check`, so judge-only comparisons can reuse the
+original transcription without uploading or transcribing the audio again.
+
 ## Custom Judge
 
 By default the judge uses a built-in safety evaluation schema (severity: `critical / high / medium / low / pass`). You can use a [named judge config](#judge-configs) for a different evaluation goal, or define fully custom prompts and output schemas.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "simpleaudit"
-version = "0.4.0"
+version = "0.4.2"
 description = "Lightweight AI Safety Auditing Framework"
 readme = "README.md"
 license = {text = "MIT"}
@@ -27,6 +27,7 @@ requires-python = ">=3.11,<3.14"
 dependencies = [
     "any-llm-sdk>=1.14.0",
     "fsspec>=2023.1.0",
+    "httpx>=0.27",
     "tqdm>=4.66.0",
 ]
 

simpleaudit/__init__.py

@@ -37,6 +37,7 @@
 from .model_auditor import ModelAuditor, ScenarioExecution
 from .auditor import Auditor
 from .targets import (
+    ASRTarget,
     CallableTarget,
     DecisionTarget,
     HTTPAppTarget,
@@ -96,6 +97,7 @@
     "HTTPAppTarget",
     "CallableTarget",
     "DecisionTarget",
+    "ASRTarget",
     "AuditResults",
     "AuditResult",
     "get_scenarios",

simpleaudit/experiment.py

@@ -331,6 +331,7 @@ async def _run_single_rep(
         on_turn: Optional[OnTurn] = None,
         audit_run_id: Optional[str] = None,
         trace_correlation: Optional[Any] = None,
+        evidence_resolver: Optional[Callable] = None,
     ) -> AuditResults:
         """Execute one rep with auto-retry on ERROR. Returns the final result."""
         attempts = 1 + self.max_retries_per_rep
@@ -348,6 +349,7 @@ async def _run_single_rep(
                 on_turn=on_turn,
                 audit_run_id=audit_run_id,
                 trace_correlation=corr,
+                evidence_resolver=evidence_resolver,
             )
             if not any(r.severity == "ERROR" for r in result):
                 break
@@ -370,6 +372,7 @@ async def run_scenario_reps(
         on_turn: Optional[OnTurn] = None,
         audit_run_id: Optional[str] = None,
         trace_correlation: Optional[Any] = None,
+        evidence_resolver: Optional[Callable] = None,
     ) -> List[AuditResult]:
         """Run a single scenario N times for one model.
 
@@ -394,6 +397,10 @@ async def run_scenario_reps(
             trace_correlation: Optional :class:`TraceCorrelation` shared
                 across reps; each rep records its ``turn_id -> trace_id``
                 links so the caller can fetch per-rep trace evidence.
+            evidence_resolver: Optional callable invoked after target
+                execution and before judge in each rep. May return a span
+                list or a dict with ``selected_spans`` + ``agentic_evaluation``
+                for agentic-aware judging.
 
         Returns:
             List of :class:`AuditResult`, one per completed rep. May be
@@ -447,6 +454,7 @@ async def run_scenario_reps(
                 on_turn=on_turn,
                 audit_run_id=audit_run_id,
                 trace_correlation=trace_correlation,
+                evidence_resolver=evidence_resolver,
             )
 
             # Persist

simpleaudit/judges/__init__.py

@@ -98,6 +98,7 @@
 from .helsedir_sexhealth_no import HELSEDIR_SEXHEALTH_NO_JUDGE
 from .helsedir_sexhealth_no_rag import HELSEDIR_SEXHEALTH_NO_RAG_JUDGE
 from .binary_abstention import BINARY_ABSTENTION_JUDGE
+from .asr_faithfulness import ASR_FAITHFULNESS_JUDGE
 from .checklist import CHECKLIST_JUDGE
 from .choice_match import CHOICE_MATCH_JUDGE
 from .groundedness import GROUNDEDNESS_JUDGE
@@ -119,6 +120,7 @@
     "helsedir_sexhealth_no_rag":  HELSEDIR_SEXHEALTH_NO_RAG_JUDGE,
     "binary_abstention":          BINARY_ABSTENTION_JUDGE,
     "checklist":                  CHECKLIST_JUDGE,
+    "asr_faithfulness":            ASR_FAITHFULNESS_JUDGE,
     "choice_match":               CHOICE_MATCH_JUDGE,
     "groundedness":               GROUNDEDNESS_JUDGE,
     "fact_check":                 FACT_CHECK_JUDGE,