Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/dependency-licenses.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,12 @@ jobs:
with:
node-version: "24"

- name: Set up Go
uses: actions/setup-go@v6
with:
go-version: "1.26.x"
cache: false

- name: Check licenses
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
Expand Down Expand Up @@ -70,6 +76,10 @@ jobs:
uv-script) uv lock --script "$script" --check ;;
npm) npm ci --ignore-scripts --no-audit --no-fund ;;
cargo) cargo metadata --locked --format-version 1 > /dev/null ;;
go)
GOTOOLCHAIN=local GOWORK=off go mod tidy -diff
GOTOOLCHAIN=local GOWORK=off go mod verify
;;
*) echo "Unknown package manager: $manager" >&2; exit 1 ;;
esac
)
Expand Down
47 changes: 30 additions & 17 deletions docs/development/dependency-licenses.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,40 +22,48 @@ The checker searches the complete tracked Git tree, regardless of folder. It
reads files from the committed base and head revisions, so untracked, ignored,
and uncommitted files are outside the check.

Supported lockfiles are:
Supported dependency inventories are:

- Python: `uv.lock` and uv script lockfiles (`*.py.lock`).
- JavaScript: npm v2/v3 `package-lock.json`.
- Rust: `Cargo.lock`.
- Go: `go.mod`, with `go.sum` required when it declares external modules.

The inventory includes external packages declared directly by each project,
including development, optional, and platform-specific direct dependencies.
Transitive packages and first-party project or workspace packages are excluded.
For Go, requirements marked `// indirect` are transitive. Versioned `replace`
directives identify the replacement module that is checked; local-directory
replacements are repository-controlled source and are excluded.

For a pull request, a package is checked when its locked name, version, source,
or lockfile location is added or changed. Removed packages do not fail the
check. The report still includes unchanged current direct dependencies so that
every run provides a repository-wide view. Only additions and changes affect
the status of an ordinary pull request.

Changed `pyproject.toml`, `package.json`, `Cargo.toml`, and PEP 723 inline-script
metadata must have a corresponding lockfile or belong to a declared locked
workspace. The workflow asks the native package manager to confirm that each
affected lock is current:
Changed `pyproject.toml`, `package.json`, `Cargo.toml`, `go.mod`, and PEP 723
inline-script metadata must have the corresponding dependency files or belong
to a supported declared workspace. The workflow asks the native package manager
to confirm that each affected dependency inventory is current:

- uv projects: `uv lock --check`
- uv inline scripts: `uv lock --script SCRIPT --check`
- npm projects: `npm ci --ignore-scripts --no-audit --no-fund`
- Cargo projects: `cargo metadata --locked`
- Go modules: `go mod tidy -diff`, followed by `go mod verify`

The npm command installs locked packages without lifecycle scripts. Cargo
metadata does not build project code. A compatible constraint edit does not
need to rewrite a lockfile when the native check accepts it.
metadata and the Go module commands do not build or run project code. Go
validation disables workspace discovery and automatic toolchain downloads so
that each module is checked independently with the CI-pinned toolchain. A
compatible constraint edit does not need to rewrite a lockfile when the native
check accepts it.

The checker does not inventory vendored code, datasets, model weights,
container or system packages, unsupported package managers, or dependencies
that are absent from a supported lockfile. A passing check is therefore not a
complete legal clearance.
that are absent from a supported dependency inventory. A passing check is
therefore not a complete legal clearance.

## Policy and unresolved results

Expand All @@ -66,9 +74,12 @@ outages are unresolved rather than being treated as successful lookups. An
unresolved addition or change fails the pull request; an unresolved unchanged
dependency contributes no license to the report.

Public metadata comes from PyPI, npm, or crates.io. Private registries, Git
dependencies, and other unsupported source forms require reviewed, exact-source
clarification.
Public metadata comes from PyPI, npm, crates.io, or the stable deps.dev API. Go
license metadata covers exact public module versions known to proxy.golang.org;
when deps.dev reports multiple licenses, every reported license must satisfy the
policy because their relationship is unspecified. Private modules, private
registries, Git dependencies in other ecosystems, and other unsupported source
forms require reviewed, exact-source clarification.

If metadata is incomplete, a maintainer can add a `[[clarification]]` entry with
`ecosystem`, `name`, `version`, `source`, `license`, and an HTTPS `evidence` URL.
Expand All @@ -86,8 +97,8 @@ itself from changes made in the same pull request.
The workflow publishes the `Check dependency licenses` status and uploads a
compact `dependency-licenses.json` artifact with three fields:

- `folders` maps every scanned project folder to the lockfiles inspected there
and its direct-dependency licenses.
- `folders` maps every scanned project folder to the dependency inventories
inspected there and its direct-dependency licenses.
- `licenses` maps every unique reported license value to the project folders
that depend on it.
- `failures` maps affected project folders to the licenses that fail the check.
Expand All @@ -96,9 +107,11 @@ compact `dependency-licenses.json` artifact with three fields:
The report contains no package-level records and does not list transitive
dependency licenses.

The lockfiles identify each direct dependency and its exact resolved version.
The reported license value comes from that version's package-registry metadata
or a reviewed policy clarification, not from the lockfile itself.
The dependency inventory identifies each direct dependency and its exact
resolved version. For Go, `go.mod` provides the version selection while `go.sum`
provides integrity hashes; `go.sum` is not treated as a lockfile or package
list. The reported license value comes from that version's public metadata or a
reviewed policy clarification, not from the dependency file itself.

For a failure:

Expand Down
125 changes: 117 additions & 8 deletions scripts/check_dependency_licenses.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

import argparse
import json
import shlex
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
Expand All @@ -24,16 +25,19 @@
import tomllib

POLICY_PATH = ".github/dependency-license-policy.toml"
LOCK_NAMES = {"uv.lock", "package-lock.json", "Cargo.lock"}
GO_MODULE_SOURCE = "https://proxy.golang.org"
LOCK_NAMES = {"uv.lock", "package-lock.json", "Cargo.lock", "go.mod"}
MANIFEST_LOCKS = {
"pyproject.toml": "uv.lock",
"package.json": "package-lock.json",
"Cargo.toml": "Cargo.lock",
"go.mod": "go.sum",
}
MANIFEST_MANAGERS = {
"pyproject.toml": "uv",
"package.json": "npm",
"Cargo.toml": "cargo",
"go.mod": "go",
}


Expand All @@ -45,9 +49,74 @@ class Dependency:
source: str


def go_mod_requirements(content: str) -> list[tuple[str, str, bool]]:
"""Return effective external Go requirements and whether each is indirect."""
requirements = []
replacements: dict[tuple[str, str], tuple[str, str] | None] = {}
block = ""
for line_number, raw_line in enumerate(content.splitlines(), 1):
code, _, comment = raw_line.partition("//")
try:
lexer = shlex.shlex(code, posix=True, punctuation_chars="()=>")
lexer.commenters = ""
lexer.whitespace_split = True
tokens = list(lexer)
except ValueError as error:
raise ValueError(f"go.mod:{line_number}: {error}") from error
if not tokens:
continue
if block:
if tokens == [")"]:
block = ""
continue
directive, values = block, tokens
else:
directive, values = tokens[0], tokens[1:]
if values == ["("]:
if directive not in ("require", "replace"):
continue
block = directive
continue
if directive == "require":
if len(values) != 2:
raise ValueError(
f"go.mod:{line_number}: require needs a module and version"
)
requirements.append((values[0], values[1], comment.strip() == "indirect"))
elif directive == "replace":
if "=>" not in values:
raise ValueError(f"go.mod:{line_number}: replace is missing =>")
separator = values.index("=>")
old, new = values[:separator], values[separator + 1 :]
if len(old) not in (1, 2) or len(new) not in (1, 2):
raise ValueError(f"go.mod:{line_number}: invalid replace directive")
replacements[(old[0], old[1] if len(old) == 2 else "")] = (
(new[0], new[1]) if len(new) == 2 else None
)
if block:
raise ValueError(f"go.mod: unterminated {block} block")

result = []
for name, version, indirect in requirements:
replacement = replacements.get((name, version), replacements.get((name, "")))
if replacement is None and (
(name, version) in replacements or (name, "") in replacements
):
continue # A local replacement is repository-controlled source.
if replacement:
name, version = replacement
result.append((name, version, indirect))
return result


def inventory(path: str, content: str) -> set[Dependency]:
"""Include every locked external package, irrespective of groups or platform."""
result = set()
if Path(path).name == "go.mod":
return {
Dependency("go", name, version, GO_MODULE_SOURCE)
for name, version, _ in go_mod_requirements(content)
}
if Path(path).name == "package-lock.json":
lock = json.loads(content)
if lock.get("lockfileVersion") not in (2, 3) or "packages" not in lock:
Expand Down Expand Up @@ -102,6 +171,12 @@ def inventory(path: str, content: str) -> set[Dependency]:

def direct_inventory(path: str, content: str) -> set[Dependency]:
"""Return locked packages declared directly by the project or script."""
if Path(path).name == "go.mod":
return {
Dependency("go", name, version, GO_MODULE_SOURCE)
for name, version, indirect in go_mod_requirements(content)
if not indirect
}
lock = (
json.loads(content)
if Path(path).name == "package-lock.json"
Expand Down Expand Up @@ -275,6 +350,29 @@ def package_license(dependency: Dependency) -> str:
if info.get("dist", {}).get("tarball") != dependency.source:
raise ValueError("registry tarball does not match locked source")
return info.get("license") or ""
if dependency.ecosystem == "go":
if dependency.source != GO_MODULE_SOURCE:
raise ValueError(
"unsupported Go source; provide an exact-source clarification"
)
info = fetch_json(
f"https://api.deps.dev/v3/systems/GO/packages/{name}/versions/{version}"
)
key = info.get("versionKey", {})
if (
key.get("system") != "GO"
or key.get("name") != dependency.name
or key.get("version") != dependency.version
):
raise ValueError("deps.dev response does not match requested Go module")
licenses = info.get("licenses", [])
if not isinstance(licenses, list) or any(
not isinstance(value, str) or not value.strip() for value in licenses
):
raise ValueError("invalid deps.dev license metadata")
if len(licenses) == 1:
return licenses[0]
return " AND ".join(f"({value})" for value in licenses)
if dependency.source not in (
"registry+https://github.com/rust-lang/crates.io-index",
"sparse+https://index.crates.io/",
Expand Down Expand Up @@ -365,6 +463,8 @@ def manifest_dependencies(name: str, data: dict) -> list:
"target",
)
] + [data.get("workspace")]
if name == "go.mod":
raise AssertionError("Go module dependencies are parsed from source text")
return [
data.get(key)
for key in (
Expand Down Expand Up @@ -466,17 +566,24 @@ def check_manifest_coverage(
if manifest.name not in MANIFEST_LOCKS:
continue
content = git_text(root, "show", f"{head}:{path}")
parse = json.loads if manifest.name == "package.json" else tomllib.loads
dependencies = manifest_dependencies(manifest.name, parse(content))
if manifest.name == "go.mod":
parse = None
dependencies = [go_mod_requirements(content)]
else:
parse = json.loads if manifest.name == "package.json" else tomllib.loads
dependencies = manifest_dependencies(manifest.name, parse(content))
lock_name = MANIFEST_LOCKS[manifest.name]
direct_lock = (manifest.parent / lock_name).as_posix()
previous_dependencies = (
manifest_dependencies(
if path not in base_paths:
previous_dependencies = []
elif manifest.name == "go.mod":
previous_dependencies = [
go_mod_requirements(git_text(root, "show", f"{base}:{path}"))
]
else:
previous_dependencies = manifest_dependencies(
manifest.name, parse(git_text(root, "show", f"{base}:{path}"))
)
if path in base_paths
else []
)
if (
not any(dependencies)
and not any(previous_dependencies)
Expand Down Expand Up @@ -508,6 +615,8 @@ def check_manifest_coverage(
if manifest.name == "package.json"
else tomllib.loads(workspace_content)
)
if manifest.name == "go.mod":
continue # Go workspaces still keep dependency files per module.
if manifest.name == "package.json":
members = workspace.get("workspaces", [])
elif manifest.name == "pyproject.toml":
Expand Down
Loading
Loading