Add RFC-0009: Skill Tracing - #37
Conversation
Placeholder for the skill tracing RFC. Tracing was drafted as part of RFC-0008 (MVP Skill Registry) and deferred out of that RFC to keep the MVP focused on the registry. This reserves the 0009 slot and carries forward the open questions raised during RFC-0008 review. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fill in the Summary, Motivation, and Open questions sections of the skill tracing RFC, restoring and reworking the tracing content deferred from RFC-0008. Eight user journeys cover the three instrumentation paths (explicit, agent framework, harness) and the analysis workflows they enable. Harness installation is now in scope for this RFC. Detailed design and remaining sections are TBD. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Status as of 2026-08-23: per reviewer guidance on RFC-0008, this draft currently contains the summary, problem statement, and user journeys so the workflows can be agreed on before technical detail. Detailed design and the remaining sections are marked TBD, and they will not be worked on until all the key stakeholders have agreed on the summary, problem statement, and user journeys. Remaining work:
Posted by Bill Murdock with assistance from Claude Code. |
Resolve three open questions based on research into current MLflow internals and framework documentation: - Skill trace queries extend search_traces filter syntax rather than adding a skill-specific function; exact matching on span attributes becomes a store-level requirement. - Automatic context capture follows the pattern MLflow already uses for other registry entities: resolving a skill records its identity, and autologgers pick it up when spans are created. - Harnesses split into two tiers: those with MLflow-provided tracing integrations get full SKILL spans; those that export OpenTelemetry natively get their emitted spans annotated when the skill invocation is identifiable, with no spans created on their behalf. ADK is recategorized accordingly: MLflow traces it by receiving its native OpenTelemetry output, so it is handled like the equivalent harnesses rather than via in-process instrumentation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
State both decisions in the journeys where they apply: the skill context manager manages its span internally and does not expose it to the caller, and the install-time record is per project or per user with project entries taking precedence. Remove both from Open questions, leaving OTel alignment and digest-based linking. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Status update: five of the seven open questions are now resolved in the draft, based on research into current MLflow internals and framework documentation.
Two questions remain open for discussion: OTel alignment (whether the Posted by Bill Murdock with assistance from Claude Code. |
Bringing harness installation into this RFC is a significant undertaking. Add an open question weighing full installation against a narrower tracing-activation command that records the coordinates of a skill the user installed themselves, and soften the summary's scope statement to point at that discussion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| ```python | ||
| from mlflow import MlflowClient | ||
|
|
||
| version = MlflowClient().get_skill_version_by_alias( |
There was a problem hiding this comment.
Could we skip the HTTP call and just put alias in the mlflow.genai.skill_context() call? I don't think you actually need to resolve the skill for the trace to link it.
There was a problem hiding this comment.
Done. link_skill() now accepts alias directly, so the caller writes no resolution call. I kept one property from the original design: the alias is resolved when the link is recorded and the trace stores the concrete version, which follows RFC-0008's convention that aliases are accepted as input everywhere but never stored in place of versions (the same rule its agent plugin member rows and alias URIs follow). A trace that stored @production would change meaning whenever the alias is repointed, which would undermine the deprecation and comparison journeys. So the lookup still happens under the hood for alias callers, but only for them, and only at link time.
Posted by Bill Murdock with assistance from Claude Code.
| the trace tree annotated with their registry coordinates, and the | ||
| skill name links to the skill's registry detail page. When the | ||
| recorded coordinates do not resolve (the version was deleted, or the | ||
| trace came from a different workspace), the UI shows a "not found in |
There was a problem hiding this comment.
A "Skills" tab on the trace like we have for "Links" would be nice.
There was a problem hiding this comment.
Added. The first journey's UI path now shows a "Skills" tab on the trace view alongside the existing linked-entity tabs, listing the linked skill versions with each linking to its registry detail page. It fits naturally now that the design records skills as trace-level links, the same shape the Links tab already renders.
Posted by Bill Murdock with assistance from Claude Code.
| they do not capture is which governed, versioned skill was active during | ||
| any part of the run. | ||
|
|
||
| This RFC adds a `SKILL` span that carries registry coordinates |
There was a problem hiding this comment.
I’m not convinced SKILL should be a new span type. The RFC describes a skill as ambient context, while a span conventionally represents an operation with meaningful start/end, status, input, and output. I imagine that in many harnesses we can observe loading of a skill but cannot reliably determine when the skill stops influencing later calls, so parenting LLM/tool spans beneath it may assert stronger causality than the instrumentation can honestly record.
Could this follow the existing prompt pattern instead where it links the concrete skill version to the trace for governance and related-trace queries? We could also annotate the currently active or activation span when that is detected. For example, if a tool call loads the skill, we can add a linked attribute on the tool call span.
This better aligns with OTel's guidance on what a span should be:
https://opentelemetry.io/docs/specs/semconv/how-to-write-conventions/#defining-spans
There was a problem hiding this comment.
I'll admit I resisted this one. My mental model of skills is that they're closer to subroutines than to ambient context: a defined invocation with a beginning and an end. Some harnesses partly support that model: Claude Code delimits skill-driven turns and its forked skills are genuinely bounded operations, and Cursor scopes a manually invoked skill to a single message. But I looked at how activation is described across the rest of the ecosystem, and it leans your way: the Agent Skills spec defines activation as a one-way load with no counterpart, and for most harnesses I found an observable activation but nothing that marks the end of a skill's influence. I wouldn't claim that's conclusive (much of it is reading documentation, and silence isn't proof), but it's enough that the RFC shouldn't build its core model on a boundary most harnesses may not have.
So the latest revision adopts your approach: a trace-level link to the skill version following the prompt-linking pattern, plus an annotation on the activation span where one is identifiable. No new span type. The rejected span design is documented under Alternatives.
Posted by Bill Murdock with assistance from Claude Code.
| ```bash | ||
| mlflow autolog claude | ||
| ``` | ||
| 3. Run the agent. When the harness loads an installed skill during a |
There was a problem hiding this comment.
I don't think Claude MLflow hook will have that level of detail. Is the plan to parse tool call outputs for any indication that a tool read a skill file (e.g. cat <skill path>) from MLflow?
There was a problem hiding this comment.
It does, essentially in the way you suggest. The MLflow Claude Code plugin already works by parsing the session transcript from its Stop hook, and skill activations are visible there today: a model-invoked skill appears as a Skill tool call whose input carries the skill name, and the tool result carries the resolved command name (the parser already keys on that field to filter skill content injection out of prompt detection). A user-invoked /skill-name takes a different shape (a command expansion with no Skill tool call), so the parser needs handling for that path too, which the detailed design will cover. And a skill run with context: fork already surfaces as a tool span with the subagent's spans nested under it. The revision states the detection approach in the harness journey: harness-native signals where they exist, and otherwise matching tool calls that read a skill's SKILL.md from its installed location, which is your cat <skill path> case.
Posted by Bill Murdock with assistance from Claude Code.
| trace came from a different workspace), the UI shows a "not found in | ||
| registry" indicator rather than failing to render the span. | ||
|
|
||
| #### Trace skills loaded by an agent framework |
There was a problem hiding this comment.
It'd be nice if we can detect tool calls that use a script bundled in a skill and annotate the tool call span as having used the skill. This should be possible by matching the paths in the tool call to where MLflow installed skills.
There was a problem hiding this comment.
Adopted. The harness journey now specifies exactly this: tool calls that use a skill's bundled files, such as a script under the skill's directory, are annotated as skill usage by matching paths against the locations recorded at install time. The same location matching also serves as the activation-detection fallback (a tool call reading a skill's SKILL.md), so both ride one mechanism, and it has the property that it never fabricates usage: a path match is positive evidence the skill's files were touched.
Posted by Bill Murdock with assistance from Claude Code.
| for spans it directly dominates) or inclusive is a design decision, not | ||
| a user choice. | ||
|
|
||
| #### Trace skill lineage to evaluation results |
There was a problem hiding this comment.
The experiment UI shows "prompts" associated with the experiment. Should we consider adding a skills tab too?
There was a problem hiding this comment.
Added. The benchmark evaluation journey's UI path now includes a "Skills" tab on the experiment page, listing the skill versions linked from the experiment's traces, following the existing prompts tab.
Posted by Bill Murdock with assistance from Claude Code.
| for spans it directly dominates) or inclusive is a design decision, not | ||
| a user choice. | ||
|
|
||
| #### Trace skill lineage to evaluation results |
There was a problem hiding this comment.
Will the UI show the diff of skills used when comparing to evaluation runs?
There was a problem hiding this comment.
Yes, added. The run-comparison view shows a diff of the linked skill versions alongside the metric deltas, so a change in outcomes is read against the change in skill configuration. It appears in both comparison journeys: the benchmark one, and a new journey comparing skill versions on production traffic, where one evaluation run is produced per skill version from the linked production traces and the diff makes the comparison legible.
Posted by Bill Murdock with assistance from Claude Code.
| APIs. Filtering evaluation results directly by skill version, without | ||
| the intermediate trace lookup, is not part of this RFC. | ||
|
|
||
| #### Detect a regression after a skill update |
There was a problem hiding this comment.
I'd suggest consolidating this user journey with the one above.
There was a problem hiding this comment.
Done. The lineage and regression journeys are merged into a single "Evaluate and compare skill versions on a benchmark" journey, which carries the lineage steps (evaluation result to linked skill versions and back) inside the comparison flow.
Posted by Bill Murdock with assistance from Claude Code.
| @@ -0,0 +1,518 @@ | |||
| # RFC 0009: Skill Tracing | |||
There was a problem hiding this comment.
Could you please make the document more concise? Have Claude determine the lowest value sections to cut. The user journeys section may be too exhaustive.
There was a problem hiding this comment.
The revision trims throughout: the token-cost journey is dropped, lineage and regression are merged, the span-model justification in the summary is reduced to a statement of the design, and the nesting material is gone. One journey was added (comparing skill versions on production traffic, which I think earns its place as the production counterpart of the benchmark journey), and the document is still net shorter than before. If it still reads as too long, pointers to whichever sections feel lowest-value would help me cut further.
Posted by Bill Murdock with assistance from Claude Code.
|
|
||
| 1. Install the skill into the harness through MLflow: | ||
| ```bash | ||
| mlflow skills install --skill-uri skills:/code-review@production \ |
There was a problem hiding this comment.
Should could probably use a bit more details. It's a bit overlooked in the current RFC.
There was a problem hiding this comment.
Expanded. The harness journey's run step now spells out what the autologger does: it identifies skill activations in the recorded conversation using the harness's own signals where they exist (a dedicated skill tool call, a slash-command invocation), falls back to matching tool calls that read a skill's SKILL.md from its installed location, annotates tool calls that use a skill's bundled files by the same location matching, and links using the install record with no registry call during the run.
Posted by Bill Murdock with assistance from Claude Code.
Replace the SKILL span model with trace-level links plus activation-span annotation, following the prompt-linking pattern, per review discussion on the span-vs-link question. skill_context() becomes link_skill(), accepting version or alias with aliases resolved at link time per RFC-0008 convention. Skill queries use a skill filter over linked entities, mirroring the prompt filter, with name-only and organization-qualified forms as proposed extensions. The rejected span design is documented under Alternatives. Journeys are consolidated from eight to seven: the token-cost journey is dropped, lineage and regression merge into a benchmark evaluation journey, the deprecation journey generalizes to security exposure, and a new journey compares skill versions on production traffic. Harness detection is specified as a ladder: native skill signals, SKILL.md read matching, and bundled-file matching for usage annotation. UI paths add a trace-view Skills tab, an experiment Skills tab, and a linked-skill diff in run comparison. Additional review fixes: organization on link_skill and filters, the OTel receiver path requires coordinates in emitted spans, exposure queries name their locations, and digest queries resolve through the registry index. New open question on locally modified installs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
Summary
RFC-0009 proposes Skill Tracing: a
SKILLspan that links MLflow traces to the registered skill versions that were active when the trace was produced, so that adoption, deprecation impact, cost attribution, and regressions can be evaluated per skill.Tracing was originally drafted as part of RFC-0008 (MVP Skill Registry, PR #26) and deferred out of that RFC. It now gets its own RFC rather than a follow-on PR.
The RFC covers three instrumentation paths: explicit (
skill_context(), with a plain OpenTelemetry equivalent), automatic in agent frameworks (LangGraph, ADK, and others), and automatic in harnesses (Claude Code and others) via an install-time record. Harness installation commands are in scope to support that third path. User journeys span instrumentation, adoption measurement, deprecation impact, token cost attribution, evaluation lineage, and regression detection.Numbering note
Extended Skill Bundles, previously RFC-0009 in PR #27, has been renumbered to RFC-0010. Skill tracing is the more urgent piece of work, so it takes the 0009 slot. The forward references in the merged RFC-0008 that point to "RFC-0009: Extended Skill Bundles" still link to PR #27 and will be corrected in a separate PR.
Posted by Bill Murdock with assistance from Claude Code.