RFC-0008: MVP Skill Registry

[jwm4]

Summary

Add a Skill Registry to MLflow: a governed, metadata-first registry for AI agent skills.

This is a restructured Phase 1 of the work originally proposed in PR #10, narrowed per Slack discussion with Databricks maintainers to focus on skills and skill bundles only.

• Skills: SKILL.md directories with versioning, lifecycle, aliases, tags
• Skill bundles: versioned collections grouping related skills
• Pull semantics: harness-agnostic content fetch from Git, OCI, ZIP, MLflow artifact storage
• Package manager integration: delegates harness-specific installation to APM, Lola, or other plugins
• Trace integration: mlflow.skill_context() creates SKILL spans with registry coordinates

Phase 2 (RFC-0009, separate PR) will extend bundles with subagents, hooks, and MCP server references.

Changes from PR #10

• Removed subagent and hook entity types (deferred to Phase 2)
• Skill bundles contain only skills (no MCP server cross-references)
• Replaced custom harness adapters with package manager plugin interface
• Removed import command and lock file format
• Aligned with RFC-0004 (MCP Registry) patterns throughout
• All design decisions from the PR RFC-0008 and RFC-0009: Skill Registry and Harness Integration #10 review are preserved

Files

• rfcs/0008-mvp-skill-registry/0008-mvp-skill-registry.md: Main RFC
• rfcs/0008-mvp-skill-registry/implementation-details.md: DB schema, store interface, REST API, SDK, CLI, package manager plugin interface

Test plan

• Review main RFC for completeness and consistency with RFC-0004
• Review implementation details for store/SDK/REST completeness
• Verify all PR RFC-0008 and RFC-0009: Skill Registry and Harness Integration #10 settled decisions are preserved
• Confirm no subagent/hook/MCP references leaked into Phase 1

Posted by Bill Murdock with assistance from Claude Code.

[HumairAK]

and warns about subagents, hooks, and MCP configurations that are not registered
in Phase 1.

I see these warnings are provided in various cases (like import). It feels unnecessary because it is expected that a skill api / option would register only skills.

[jwm4]

I could argue it either way, but Yuki specifically asked for the warnings, and they seem somewhat useful to me.

[HumairAK]

It would be nice if the RFC also points out the plan for the others in a future enhancement section, which, if any, should be prime candidates for auto log support.

[jwm4]

Broadly speaking, I think the process would be something like this:

1. Start with the list of harness agents that have auto-loggers at https://mlflow.org/docs/latest/genai/tracing/integrations/#coding-agents--long-running-agents
2. Prioritize based on popularity, openness, importance to whatever contributor wants to do the work, etc.

In addition to supporting the harness agents, you might want to support the SDKs at https://mlflow.org/docs/latest/genai/tracing/integrations/#agent-frameworks-python and https://mlflow.org/docs/latest/genai/tracing/integrations/#agent-frameworks-typescript . Not all of those have a "skill" concept, and you can't autolog skill use if the SDK that you're using doesn't have a concept of a skill. (You could still log it yourself if you layer a skill concept on top of the SDK you're using of course).

This seems like a long, hard conversation to have to get any sort of meaningful conclusions. I'd prefer to keep it out of the RFC because I'm a little concerned that resolving this debate would take a bunch of time that would slow down work on this RFC in order to pre-decide things for future enhancements. Furthermore, I think a lot of things will change between now and the time that someone starts working on future enhancements, so whatever we decide now might turn out to be irrelevant by that time anyway.

[HumairAK]

One thing that caught me by surprise was the decision to have MLflow locally materialize the skill or skull-bundle. I think this is a good way to ensure MLflow retains domain ownership over resolution semantics, instead of trying to squeeze registry behavior into another tool's package-resolution abstraction.

I think it would be good to explicitly mention the justification somewhere.

I'd imagine there are some tradeoffs here as well, for example if a package manager has features / capabilities around remote source semantics. If you agree, consider adding a small note about it.

[jwm4]

Yes, I agree this needs some more thought. We'll sort it out in the upcoming installation RFC now that we've decided to defer installation.

[mprahl]

Looks great! I left a few comments.

[mprahl]

Could we reconsider the "every version of a given agent plugin must be the same kind" requirement? I think each version should be consistent but I don't see why we'd restrict a user from initially registering a monolithic bundle and then later to use an assembled approach without having to tell everyone to update their lock files to use a new agent plugin name.

[jwm4]

Good point, relaxing this. Kind is now resolved per version and no longer required to be consistent across an agent plugin's versions: a plugin can be packaged in one version and assembled in another under a stable name, so an author can migrate authoring styles without forcing consumers onto a new name and new pinned references. This is safe because nothing aggregates kind across versions. Pull, latest-resolution, and member sourcing each read the kind of the specific version they operate on. One caveat I spelled out: a name-only pull resolves to the latest version, so it can change behavior across a kind boundary (packaged fetches the whole package as a unit; assembled fetches each member individually), while a version-pinned reference is unaffected.

One safeguard I made explicit in the process: because an assembled version can reference a standalone skill the caller does not own, and a later packaged import could then discover the same name, re-import now states that appending a version to a matched existing skill requires EDIT on that skill, evaluated separately from plugin permissions (the same rule as any version creation). So a reference-level relationship can never be laundered into a write on someone else's skill. Updated both the main RFC and implementation-details.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

Could we raise an error instead when someone tries to delete a skill referenced by a plugin version? I think this would be less surprising as a user may not be aware that their skill is being used by a plugin version.

[jwm4]

Agreed, erroring is less surprising. This now falls out of a referential-integrity check that runs before anything is removed: a skill version referenced by a live (non-deleted) agent plugin version cannot be hard-deleted while that reference exists, and this applies to delete_skill on a member skill directly, not just to the plugin-side cascade. The delete fails atomically with nothing removed, and the operator clears the other reference and retries. References held only by soft-deleted plugin versions do not block the delete. This replaces the earlier behavior of deleting most members and silently keeping the referenced ones. See 0008-mvp-skill-registry.md around the top-level delete section.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

Is source not required? When would it not be set.

[jwm4]

source is optional. It holds the typed pointer for external content, so it is set for git, oci, and zip, and for mlflow it is the MLflow artifact tree. It is unset for an assembled agent plugin version, which has no plugin-level package at all (its members carry their own sources), and transiently while an mlflow upload is still in draft before the artifact tree exists. The field is already marked optional in the SkillVersion description, and the null case is stated where assembled plugins are defined ("an assembled plugin has no agent plugin-level source"). I can add a one-line cross-reference if you would like it spelled out in one place.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

Nit: Why not avoid creating the DB record in the first place until upload completes? If this is using the artifact repository APIs, then it should be able to delete artifacts.

[jwm4]

The draft-first ordering is deliberate: the upload destination is derived from the created version's identity, so the version record has to exist before the client knows where to upload. Keeping the version in draft until the upload completes means a failure leaves a discarded draft rather than a content-less active version. You are right that a backend with deletion support can clean the orphaned files up immediately; the drawback note only calls out that a backend without deletion support retains them until GC. I can add a sentence noting the client attempts to delete the orphaned draft artifacts on failure where the backend supports it, if that reads better, but I would keep the create-then-upload flow itself.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

I’m wondering whether import_agent_plugin needs to be a separate API. create_agent_plugin_version already feels like the natural place for plugin members, but today it only accepts skills for assembled plugins. Would it be simpler to expand version creation to also support discovered embedded members for monolithic imports, rather than introducing a separate import-specific write path?

[jwm4]

I would like to keep import_agent_plugin as a separate write path. The two operations look similar but take fundamentally different member inputs: create_agent_plugin_version takes skills as URI references to skills that already exist in the registry (the assembled case), while import takes definitions for the skills embedded in the package source, discovered during local inspection, and creates them. That difference drives the rest: import creates N new skill versions plus the plugin version atomically as one unit of work with all-or-nothing rollback, and returns both the plugin version and the newly created skill versions, whereas version creation writes a single entity referencing pre-existing skills and returns just the version. Folding them together would mean one method with mutually exclusive skills vs. member_skills parameters, a mode switch, and two different return shapes and failure models behind one signature.

The shared intent you are pointing at is already honored where it is cheap: both go through the store layer, and the high-level mlflow.genai surface presents import as a single call (it fetches the source, detects the format, translates it, then posts the prepared payload). So the surface is not as duplicative as the two primitives suggest. Keeping the two write paths distinct keeps each REST contract (POST .../versions vs. POST /import) precise about what it creates.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

Is there a reason not to set source to the resolved MLflow artifacts URI and drop source_type? The MLflow client can infer the source type from the source value.

[jwm4]

I would prefer to keep source_type as an explicit server-set field rather than inferring it from source. The value cannot disambiguate every case: an assembled plugin has no source at all, and an mlflow artifact-tree URI versus other null-source flows cannot reliably be told apart by parsing alone. We worked through exactly this earlier in the review and landed on keying the type on the creation flow / endpoint rather than on the source value, which removed that ambiguity. Inferring from source would reintroduce it for the null and MLflow cases, and it would mean the server has to parse a user-supplied URL to classify content. I would like to leave this as-is.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

It's implied but not spelled out as far as I could see, but could you make it clear that source_type of mlflow is only supported when MLflow serves artifacts?

[jwm4]

Good call, making it explicit. source_type="mlflow" stores content in MLflow artifact storage, so it is only usable where the deployment serves artifacts; a tracking-only server can still use the external source types (git/oci/zip) but not mlflow. Added a sentence to the SkillVersion source description.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

I reviewed the latest changes. My biggest concern is this comment:
#26 (comment)

The rest are mostly optional.

[mprahl]

This may be implied but it'd be nice for the MLflow client to detect skills that stayed the same and reuse those skills in the member references rather than creating a new version. This has benefits such as being able to see the diff clearly between agent plugin versions and old traces and new traces using those skills link to the same skill.

[jwm4]

Thanks Matt, this pushed me to bring the content digest back (I'd dropped it earlier as non-essential to the MVP). On import, each discovered skill still gets a new version whose source is derived from the newly imported package, and I now record the content digest on that version. It's computed client-side during local inspection, stored, returned on get, and indexed so callers can filter by it (digest = '<hex>'). When a skill's content is unchanged the new version carries the same digest as the prior one, so a client can group versions by content after the fact, giving you the clean diff between plugin versions and the "old and new traces point at the same content" linkage, on the read side.

I didn't make import literally reuse the prior version because of the one-immutable-source-per-version commitment: a newly imported packaged-plugin version containing a member that points into an older package is exactly the incoherence I'm avoiding. Minting a new version per discovered skill keeps each version with one immutable, package-derived source, and the digest carries the identity signal instead.

One caveat: the digest is client-asserted and not server-verified, so equal digests mean the registering clients asserted identical content, not that the registry guarantees byte-for-byte identity. The RFC exposes and indexes the field; it doesn't build a diff or trace-linking feature on top of it.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

I think it may be better to error on this case rather than silently keeping some skills around. It's a less surprising behavior.

[jwm4]

Agreed, erroring is less surprising. Cascade delete now checks before removing anything: if it would hit a member still referenced by a live (non-deleted) plugin version other than the one being deleted, the whole delete fails atomically and nothing is removed, and the operator clears the other reference and retries. Membership rows held only by soft-deleted plugin versions do not block the delete and are purged with the member. This replaces the earlier behavior of deleting most members and silently keeping the referenced ones.

Comment from Bill Murdock with assistance from Claude Code.

[khaledsulayman]

Wouldn't we also want this name inference to happen client-side, similarly to the plugin import flow? How are we guarding against nefarious URL submission?

[jwm4]

Yes, agreed on both counts, and that's how it now works. Name inference for skill registration happens client-side during local inspection, exactly like the plugin import flow: the client reads the skill's SKILL.md and computes the content-derived fields (name, description, keywords) locally, then submits them. The registry server never fetches a user-supplied source URL, for skill registration and plugin import alike. That is the guard against nefarious URL submission: there is no server-side fetch of an untrusted URL to exploit. I've now stated this explicitly on the skill path rather than leaving it implied from the import path.

Comment from Bill Murdock with assistance from Claude Code.

[khaledsulayman]

Should name-only member resolution for plugins ever really select a draft? I agree with the ergonomics of get/pull for individual skills, especially being that it is reevaluated at each call. Having a draft skill be frozen as a permanently pinned plugin member (if the plugin is registered prematurely) seems counterintuitive.

[jwm4]

Agreed, that would be counterintuitive. Name-only member resolution now resolves to the skill's latest active version and never pins a draft or deprecated one. Because a plugin member is frozen at registration rather than re-evaluated, if the skill has no active version the plugin create fails, directing the author to publish the skill or pin an explicit version rather than freezing a prematurely-registered draft. Individual get/pull keep the broader latest-resolution rule you mentioned, since they re-evaluate on each call.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

Could we drop finalize and just accept upload first then register approach?

[jwm4]

I went with a single-call registration instead of a finalize step. Because a skill is small, a local-path registration now carries the packaged content inline in one atomic operation, so there is no create-then-upload-then-finalize sequence and no in-flight version state. I dropped the finalized column, the finalize PATCH parameter, and the skip-non-finalized resolution rules that only existed to guard the gap between creating a version and its content arriving.

Two supporting details: version numbers now come from a durable per-identity counter that outlives a hard delete, so a number and its content path are never reused; and the registration wire format is specified as multipart/form-data (a JSON metadata part plus a gzip-tar content part) with archive validation and a size limit.

Comment from Bill Murdock with assistance from Claude Code.

[mprahl]

A couple small comments but looks good otherwise!

[mprahl]

Could we drop skill_version_counters and allocate like the model registry: MAX(version)+1, restart at 1 after the parent is gone, retry on uniqueness races? If leftover blobs are a concern, I would follow the pattern for experiments rather than an immortal counte where we keep the skill in a deleted lifecycle until mlflow gc hard-deletes it and removes skills/.../<version>/, the same way it removes experiment artifacts. Skipping a version if that prefix is non-empty could be extra caution for source_type=mlflow only.

[jwm4]

Done. I dropped the durable counter entirely and switched version allocation to MAX(version) + 1 with a retry on the (workspace, organization, name, version) primary-key uniqueness violation, matching the model registry.

Pulling that thread further simplified two more things:

• Registration is now a single multipart request that carries the content, so the two-phase upload and the finalize step are gone.
• Each upload is stored at a unique server-chosen artifact path recorded on the version's source, never derived from the version number. So a version that restarts at 1 after a hard delete can never collide with a prior incarnation's content, which was the original reason the counter needed to be immortal.

Content is reclaimed when its owning version is hard-deleted: the server commits the row deletion first, then does best-effort blob cleanup (the artifact store isn't transactional with the DB, so committing first avoids stranding a restored row on missing bytes). The rare orphaned bytes from a crashed upload are intentionally accepted as a small leak rather than swept.

Pushed in e8a3aa1.

Comment from Bill Murdock with assistance from Claude Code.