Trial: rebuild Kuber.jl on OpenAPI.jl 1.0 - #69
Draft
tanmaykm wants to merge 46 commits into
Draft
Conversation
Start of the openapi-v1-trial line (see OpenAPIv1TrialBranchPlan.md §1): - deps reduced to Dates/HTTP/JSON/OpenAPI plus the Base64/UUIDs stdlibs the new generated modules import; Downloads, TimeZones and (unused) Random gone - HTTP is now a direct dep: it activates OpenAPI's HTTP extension and the discovery probes use it directly - [sources] pins quinnj/OpenAPI.jl @ 1ff9ba8 (package version 1.0.0); to be replaced by a tag + full regeneration when PR 103 merges - floors: Julia 1.11, HTTP 2, JSON 1.7 Also checks in the evaluation notes, the trial plan, the runnable prototypes every decision was verified against, and the watch-latency probe, so the branch is self-describing. src/ still references OpenAPI.Clients/Downloads, so `using Kuber` fails cleanly here by design; Phase 1 replaces the generated layer.
Replaces src/ApiImpl/api (openapi-generator, Swagger 2.0, k8s 1.23-era) plus api_typemap.jl/api_versions.jl with 17 OpenAPI.jl 1.0 client modules generated from upstream kubernetes/kubernetes v1.35.4 OpenAPI v3 group documents, and a registry of lookup tables over them. Pipeline (gen/openapi_v1/, all inputs and outputs checked in so fetch -> patch -> generate is reproducible and auditable): - fetch_specs.sh pulls pristine group documents from a release tag and records provenance in SPECS_ORIGIN - patch_k8s_spec.jq is the middle path: nullable meta.v1.Time, meta.v1.MicroTime and every array property (Go nil slices marshal as null), and NO /watch/ path rewriting — watching goes through the non-deprecated list ops - generate.jl generates each patched document in strict mode (~30s) - emit_registry.jl emits registry.jl The registry replaces the string-munging + `eval` lookups it used to take to find an operation: GROUP_MODULES apiVersion -> module (was APIVersionMap) MODULE_GVS module -> apiVersion KIND_TYPES (apiVersion, kind) -> type (was Typedefs + kuber_type sniffing) OPS (module, verb, kind, scope) -> operation function OP_PARAMS same key -> positional argument names Two departures from the plan sketch, both forced by the specs: - OPS is keyed by module as well as (verb, kind, scope). A 3-tuple cannot express two shipped versions of one kind, and the trial ships two: autoscaling/v1 and autoscaling/v2 both define HorizontalPodAutoscaler. - OP_PARAMS exists because generated positional order is path order — the namespace comes FIRST (`readcorev1namespacedpod(namespace, name)`) and a required body comes last, the reverse of the old client. Two things the emitter learned from the specs rather than assuming: - Subresource kinds come from the parent resource path, not the operation's own x-kubernetes-group-version-kind (which names the subresource type: pods/exec is PodExecOptions, pods/eviction is policy/v1 Eviction). Parent kind plus capitalized subresource reproduces the operationId tails exactly, so :PodLog reaches the table through the generic path instead of a special case. - Generated model and function identifiers are read off the planner, not recomputed. The naming rules normalize non-identifier characters, dodge Base/Core and reserved names, and disambiguate collisions with a counter; a hand-rolled version got io.k8s.apiextensions-apiserver... wrong on the first try and would have drifted silently later. Meta kinds (Status, DeleteOptions, WatchEvent, APIResourceList) are registered under every group-version in every document and each module has its own copy, so KIND_TYPES resolves them by policy: own group-version wins, then core, then alphabetically first. Cross-module identity for those types is therefore not guaranteed — `isa(res, kind_to_type(ctx, :Status))` on a non-core response is a Phase 3/4 item. src/Kuber.jl is reduced to the generated layer for now; helpers.jl and simpleapi.jl still target the 0.2.x runtime and stay on disk as the porting reference for Phases 2 and 3. Gate (§6.1): pipeline clean, `using Kuber` precompiles in ~23s and loads in ~2.2s, test/registry.jl green (3639 assertions, no cluster needed).
Plan §4.3 asks for this before k8s_retry_cond is written, since the exception types are runtime internals rather than a stable contract. A manual tool, not part of runtests.jl; rerun it whenever the OpenAPI pin moves. Six failure modes, probed against raw TCP servers (precise control over truncated bodies and dropped connections) and the live k3s cluster. Three results contradict what the plan assumed: - the watch call never throws on consumer close. It returns at the response head, so there is no in-flight call to retry and the `isopen(stream)` guard has nothing to guard; retry-vs-stop has to live in the re-watch loop instead. - a connection dropped on an item boundary closes the channel CLEANLY, exactly like a watch ending normally. A clean close therefore cannot be read as "stop" — only the consumer closing the channel can. - 410 Gone is not an ApiError. k8s answers an expired resourceVersion with HTTP 200 and an in-stream ERROR event carrying a Status(reason=Expired). The rest confirm the plan: retryable statuses arrive as ApiError with .status, transport failures as HTTP.ConnectError (is_request_interrupted is gone), and a truncated item closes the channel with DecodeError.
Context, clients, discovery, retries, exceptions and conversions, per plan §4. KuberContext now holds a server URL and a lazily-built client per group module, because a Runtime.Client is bound to its module compiled _SPEC and cannot be shared across modules. Each client registers the watch codec at construction; the codec is inert until a call passes accept=WATCH_MEDIA, so buffered calls on the same client keep decoding typed models. Discovery keeps its semantics (probe /api and /apis, tolerate groups we do not ship, honour the `override` kwarg, preferred version first) but issues the two probes with plain HTTP.jl: they were the only reason the old code needed the generated ApisApi/CoreApi wrappers. ctx.modelapi is built from KIND_TYPES rather than a names() scan, so it holds exactly the addressable kinds, and core is registered first with earlier registrations winning — the meta kinds every group redefines (Status, WatchEvent, DeleteOptions) resolve to core deterministically instead of by dictionary order. k8s_retry_cond encodes what test/characterize_retries.jl actually observed rather than what the plan predicted. ApiError.status covers the retryable statuses; transport failures are now stated as an exclusion over HTTP.HTTPError, since HTTP 2.x has no RequestError and puts everything under that supertype — retry the accidents, not the decisions (CanceledError, StatusError, TooManyRedirects, AddressInUse, RetryDenied). The old `isopen(stream)` guard is gone from the retry path entirely: a watch call returns at the response head, so it can never be the in-flight call being retried, and stop-vs-re-watch moves to the watch loop in Phase 3. Timeouts move from the client mutable timeout[] to request_options on the context, carrying HTTP 2.x names (request_timeout, connect_timeout, read_idle_timeout, and TLS config). set_timeout sets request_timeout, and watch calls drop it: a watch has no meaningful overall deadline, and k8s bounds one with the timeoutseconds query parameter instead. Gone with the 0.2.x runtime: check_api_response and the (result, response) tuples (operations throw ApiError now), get_return_type sniffing, the Downloads.Response header handling, the Connection header and httplib selection, the int-or-string val_format extension (v3 declares IoIntOrString as a real oneOf), and the convert() piracy on String/Dict. kuber_obj now decodes through KIND_TYPES, and _field() is added for reading fields that may be ABSENT — the one user-visible semantic change, since `nothing` now means an explicit JSON null. Gate (§6.2): test/helpers.jl green offline (82 assertions), and live discovery against k3s v1.35.4 maps 16 groups and 101 kinds with the server preferred versions ordered first.
The verbs keep their signatures and semantics; internals become registry
lookups (§5). Resolution is: module from the apiversion kwarg or ctx.modelapi,
then OPS[(module, verb, kind, scope)] with the old namespaced -> cluster ->
all-namespaces fallback chain as table probes instead of isdefined probes.
Positional arguments come from OP_PARAMS, so the namespace goes first and a
required body last. Snake_case kwargs are translated at the boundary
(label_selector -> labelselector) and nothing values are dropped rather than
forwarded, since generated optionals are Union{Absent,T} and an explicit
nothing fails request validation.
The watch rewrite is shaped by the characterization, not the plan sketch. The
generated call returns at the response head, so `list` cannot delegate and
return — it would close the stream immediately and yield zero events. Instead
its watch branch pumps the raw channel inline and returns only when the watch
is over, which keeps `watch(processor, ctx, list, O)` and both of its `finally
close(stream)` blocks working exactly as they did under the 0.2.x client. In the
loop: the consumer closing the public stream is the only stop signal (#67/#68),
a clean close of the raw channel means re-watch from the last resourceVersion
seen (a dropped connection on an item boundary is indistinguishable from a
normal end), a DecodeError means the same, and an in-stream ERROR event with
code 410 restarts without a resourceVersion — which is how k8s actually reports
an expired one.
Two further spec lies surfaced, both found by exercising verbs the evaluation
never had (it only ever listed and watched). Both are new patch rules, not
validation opt-outs:
- request bodies documented as `*/*`. Every create/replace body says `*/*`,
which no client can encode to: there is no `*/*` encoder, and the runtime
would send `Content-Type: */*` even if there were. Rewritten to
application/json, which is what we send and what k8s accepts. Patch bodies
keep their five explicit media types.
- DELETE 2xx responses documented as `Status`. A delete usually answers with
the deleted object instead — verified live: deleting a Job returns the Job,
deleting a Deployment returns a Status, so the ambiguity the old client hid
behind get_return_type sniffing is real and both shapes occur. `oneOf:
[Status, resource]` was tried and rejected (the generator emits one wrapper
type per response code per media type — eight for a single delete). The
responses now carry the empty schema, which states what is actually true, and
delete! restores the type from the payload kind/apiVersion through
KIND_TYPES — the same second-stage decode watch frames use.
update! wraps the patch in the generated Patch type (an open object, so a Dict
cannot be passed through) and validates patch_type against the documented media
list, since k8s offers no plain application/json for PATCH. Custom metrics throw
a clear out-of-trial error (§0). get_logs needs no special handling: the pod-log
response decodes to a String.
Registry gains OP_BODIES (body type + documented media types per operation) so
update! reads both from build-time tables rather than reflection.
Gate (§6.2): test/simpleapi.jl green offline (59 assertions, including the #67
watch-abort canary). Live against k3s v1.35.4: put!/update!/delete! round-trip a
Pod, a Deployment and a Job; watch delivers a typed initial list plus typed
ADDED/DELETED events; get_logs returns pod logs.
runtests.jl now runs the three offline suites first (registry, helpers, simpleapi) and then the integration suite, which is skipped with a warning when no API server is reachable and honours KUBER_TEST_SERVER. Expected diffs from the 0.2.x suite, all of them semantic rather than cosmetic: - the versioned-model test used batch/v1beta1 and batch/v2alpha1 CronJobs, and the override test used apps/v1beta2 and apiregistration.k8s.io/v1beta1. None of those exist on a 1.35 server; autoscaling is now the group that still serves one kind in two versions (v2 preferred, v1 available), so it is what exercises versioned typing and the `override` kwarg. - Typedefs.CoreV1.WatchEvent -> KuberEvent, and event.object is already typed, so the kuber_obj round-trip in the watch assertions is gone. - delete! assertions compare kuber_kind, not the type: each group module has its own Status type, so a batch/v1 delete can never be `isa` core Status. - the timeout test is rewritten against request_options (there is no DEFAULT_TIMEOUT_SECS to compare with; unset means no deadline) and also asserts a watch drops request_timeout. - new coverage: all-namespaces listing, a 404 surfacing as KuberException with a decoded Status, a rejected patch media type, and a Job + Deployment create/patch/delete round-trip. - the `killall kubectl` teardown is dropped: it worked around a libcurl segfault on exit, and Downloads.jl is no longer a dependency. - test_watch_processor_failure moved to test/simpleapi.jl, where it belongs with the other offline checks. watch_latency.jl is adapted for the new API, and running it surfaced a real usability trap: metadata.annotations decodes to a generated open struct, not a Dict, so reading it by index silently failed and every MODIFIED event was reported as MISSED. This is trap 6 of the plan (index open payloads via additional_properties) meeting ordinary code. Added `kuber_props` to normalize the shapes a k8s string map can take — ABSENT/null, a real Dict, or the struct — since labels and annotations are far too common to make callers reach into additional_properties themselves. Docs: README gains a trial-branch section covering every user-visible change (ABSENT vs nothing, lowercase field names, open string maps, KuberEvent, per-module Status types, request options, what is out of scope); SupportedAPIVersions.md is rewritten for the 17 generated group versions and records what is deliberately absent; WalkThrough.md and Metrics.md get banners, and WalkThrough loses a `!== nothing` check on a field that is now ABSENT. CLAUDE.md is rewritten for the two-layer-plus-registry architecture, the watch pump, and the five patch rules. CI drops Julia 1.6 for the 1.11 floor. Gates (§6.3-§6.5): full suite green against k3s v1.35.4 — 3639 + 89 + 59 offline and 303 live assertions. The prototype cross-check k8spristine_v3.jl passes 11/11 against the pinned OpenAPI, confirming upstream behaves as the evaluation recorded. watch_latency reports 5-11 ms reaction times after warmup, so the watch wrapper adds no buffering of its own.
…p spinning test/watch_recovery.jl exercises the four watch criteria of §6 that need failure injection rather than a cooperative cluster: a consumer stopping a watch, a stream ending mid-watch (#68), a truncated item, and an expired resourceVersion. It runs offline against a fake apiserver and is included from runtests.jl. Writing it turned up three things. The pump only noticed a stop when the next frame arrived. On a quiet resource that can be minutes, which left `watch()` hanging and — under `watch(streamprocessor, ...)` — kept the @sync alive after the processor died, the exact deaf watch #67 fixed. A watcher task now closes the raw channel when the consumer closes the public stream, so a stop takes effect in ~250ms regardless of traffic. The re-watch loop could spin. k8s_retry only wraps the *establish* call, so a server that answers 200 and ends the stream without delivering anything is not a failure and nothing throttled the next attempt — reachable with an unservable resourceVersion or a proxy dropping long connections, and it would have hammered the apiserver. Consecutive establishments that deliver no events now back off 0.25s to 8s, reset by any delivered event, and abandon the wait as soon as the consumer stops. `wait(stopwatcher)` is guarded so it can never mask the error that reached the finally block. Also fixed: `get(ctx, O, name)` in a watch context clobbered a caller-supplied field_selector with its own metadata.name filter; the two are now ANDed as k8s expects. The harness is built on HTTP.jl rather than raw TCP, after a hand-rolled chunked server turned out to deliver nothing to the HTTP.jl client until the connection closed — bytes curl streamed happily. Using one library on both ends keeps the framing beyond question; the real apiserver framing stays covered live by watch_latency.jl. Handlers hold responses open by polling a flag rather than sleeping, because close(server) waits for in-flight handlers and a sleeping one hangs teardown instead of failing the test. Verified after these changes: full suite green against k3s v1.35.4 (3639 + 89 + 59 + 32 offline, 299 live) and watch_latency reports 0 missed events with 5.6-11.2ms medians, so the stop watcher and backoff cost nothing in the steady state.
A connection aborted part-way through a chunk — an apiserver restart or a network drop, the one #68 shape a clean stream end does not stand in for — closes the watch channel with an HTTP.jl error (ParseError: unexpected EOF while reading HTTP/1 data), not the DecodeError a truncated *item* gives. The pump recovered from DecodeError only and rethrew everything else, so this killed the watch outright. The earlier characterization missed it because that probe used an unframed response body, where an abort closes cleanly; real k8s is chunked. Verified against an HTTP.jl server force-closed mid-chunk. The pump now re-establishes for anything k8s_retry_cond accepts as well as DecodeError, which strictly widens recovery and cannot regress the stop path — consumer close is checked first. Asserted in test/helpers.jl at the decision point, since reproducing it end-to-end needs the listener killed underneath the client, leaving nothing for the retry to reach. OpenAPIv1TrialResults.md records what the branch actually does: the five deviations from the plan with their reasons, the §6 measurements (precompile 22s, load 0.36s, TTFX ~15s for the first list(ctx, :Pod), steady state 11.5ms of which 78% is response validation, watch reactions 5.6-11.2ms), the test suite inventory, the expected diffs in the adapted tests, and the follow-ups. The plan gets a header pointing at it, since the plan is committed on this branch and is the first thing a reviewer reads — five of its statements are now false, and commit messages are not where design truth should live. Full suite green after both changes: 3639 + 94 + 59 + 32 offline, 307 live.
…ated from CI inherited master's kind pin (v0.11.1, node image kindest/node:v1.21.1). That was harmless on the 0.2.x line but not here: the client is generated from the v1.35.4 OpenAPI documents and strict response validation checks every reply against those schemas, so the cluster version is part of the contract. The first push failed exactly one assertion on all three Julia versions -- ctx.apis[:Autoscaling][1] is the v1 module because a 1.21 apiserver prefers autoscaling/v1 (v2 went GA in 1.23). All 3824 offline assertions passed unchanged on 1.11, 1 and nightly, so the branch was fine and the workflow was not. Pin kind v0.32.0 with the v1.35.5 node image (one patch release off the spec tag; patch releases do not move the API surface) and record the CI result in the results doc, including that the 1.11 compat floor is now exercised.
…ot provide Adds OpenAPIv1ConsumerGaps.md, a working document to iterate on. Surveys JuliaRun.jl (with JuliaHubK8sApi.jl), services/JobLoops, packages/K8sReflector, JuliaRunPool, AccessControl and BillingService against what this branch provides and what its tests cover. The headline is that test coverage is the second-order problem. The consumers do not use Kuber's generated layer at all: they use the verb layer over JuliaHubK8sApi, which is a drop-in replacement for api/Kubernetes.jl + api_typemap.jl + api_versions.jl, plugged in through KuberContext(apimodule) -- a parameter this branch removed. It also covers far more group versions than the 17 upstream ones shipped here, including CRD groups and custom.metrics.k8s.io that gen/openapi_v1/ deliberately cannot produce from release-tag specs. Split into seven hard incompatibilities (C1-C7) and sixteen test gaps (G1-G16), each with a checkbox and a stable identifier to cite in review. The one silent correctness item is G1: the pump swallows the in-stream 410 and re-watches with no resourceVersion, so K8sReflector's cache-invalidation path is dead code and its store keeps phantom entries for objects deleted while the watch was gone. Mechanism verified against the code and against a live cluster. Supersedes the narrower ABSENT/open-struct checklist line in the results doc.
… real usage The first revision framed C1 as one big blocking item. It is two problems with very different costs, and the mechanism half was overstated. C1a, the mechanism, is cheap: _new_client is already duck-typed on the module (mod.Client, no _SPEC reference anywhere in src/), the verb layer binds the registry tables rather than their contents (const bindings to mutable Dicts, so merge! is visible with no recompilation), and the keys make merging conflict-free -- OPS is (module, verb, kind, scope), KIND_TYPES is (apiVersion, kind). So registration is a merge! of six dicts, not a redesign. It must run in the downstream package's __init__, since mutations to another module's state do not survive precompilation. Restoring KuberContext(apimodule) is argued against: it saves no regeneration and reinstates the per-context scoping that keying OPS by module removed. C1b, the content, is small once scoped. Adds an appendix auditing what JuliaHubK8sApi's 43 group versions are actually used for: this branch's 17 are a strict subset, and of the 26 extras only metrics.k8s.io/v1beta1 and custom.metrics.k8s.io/v1beta1 are reachable through Kuber (v1beta2 needs a runtime check). No CRD group is used at all -- JuliaRun's ServiceMonitor and Prometheus manifests are kube-prometheus YAML applied by kubectl, and its one karpenter reference is a node label string. The audit records how the dynamic put!(cm, Symbol(job["kind"]), job) path was traced, since a grep for type names would have been wrong three times over. C1c prices the JuliaRun port: Typedefs is a generated tree of plain aliases, so its ~28 references survive verbatim if the tree is re-emitted over the new type names. Only the three WatchEvent/Status isa checks have to change. C1d records the dynamic register_crd! option as deferred future-proofing rather than a prerequisite, including that discovery already supplies plural, kind and scope, and that it would bypass response validation. C5 shrinks to the only remaining JuliaHub feature gap: two group versions plus reimplementing the two custom-metrics helpers.
The 0.2.x plug point was `KuberContext(apimodule)`: a consumer handed Kuber a drop-in replacement for its whole generated layer. This branch dropped it, which is what blocks JuliaRun and JobLoops, since they reach k8s through JuliaHubK8sApi rather than through the groups Kuber ships. `Kuber.register!` replaces it by merging rather than substituting. The registry tables are const bindings to mutable Dicts and every key carries either the group version or the module, so an external layer's six tables merge in without touching, or recompiling, what is already there. Registration is process-global; resolution stays per-context, because ctx.apis and ctx.modelapi are still built at discovery from whatever the server serves. Validation runs over the whole registration before any of it is merged, against the invariants test/registry.jl asserts over the merged result, so a malformed layer cannot half-load. A group version already served by a different module is an error rather than a silent override, and re-registering identical content is a no-op. `unregister!` undoes a registration and refuses to touch the group modules Kuber ships. No registry-generation counter: both consumers load their layer with a top-level `using`, so __init__ runs before any context exists, and invalidating contexts automatically would mean changing the ctx.initialized guard two test suites set by hand to fake discovery. Tested offline against a hand-written fake group module — no spec, no server. Every registration is undone in a finally, since the tables are global and test/registry.jl asserts invariants across all of them.
The docstring claimed register! checks the invariants test/registry.jl asserts over the merged result, and two families were missing: the OP_PARAMS shape (namespace first when namespaced, absent otherwise, body last and only for verbs that send one, no repeats) and the trap that no deprecated /watch/ operation leaks in. The params ones have teeth beyond doc accuracy, because _positional consumes OP_PARAMS positionally: a mis-emitted external table would be accepted at load and fail confusingly at call time. Also fix the README snippet, which was not runnable as printed — the emitted registry names its group modules bare, so they have to be included first, and the module needs Kuber in scope. That snippet is the copy-paste target for whoever regenerates JuliaHubK8sApi.
k8s answers a watch resumed from too old a resourceVersion with an in-stream
ERROR event carrying Status(code=410). The pump consumed it and re-watched with
no resourceVersion at all, which k8s answers by replaying current state as
synthetic ADDED events. That recovers the connection but not the truth: nothing
ever mentions objects deleted while the watch was gone, so a consumer keeping a
cache holds phantom entries for the life of the process. K8sReflector's 410
handler — which invalidates its store — was dead code against this branch.
The pump now lists again on expiry, pushes that list onto the stream, and
watches from its resourceVersion, which is what client-go's reflector does. It
removes the phantom-entry class rather than delegating it: a consumer that
replaces its store on a list frame is correct without knowing expiry exists.
The contract that makes it one rule rather than two: a list object on the stream
means complete current state. It was already the first frame; now it is also the
resync frame, so a consumer written against the initial list already handles it.
A KuberEvent("RESYNC", …) marker was considered and rejected — two shapes for
"here is full state" is worse than one, and on the get path (whose initial frame
is a single object) a resync-typed event would be stranger than a plain list.
The 410 ERROR frame is still not delivered: the list carries strictly more than
the Status did, and surfacing an error for a condition Kuber recovers from would
break consumers that treat ERROR as fatal.
push_initial=false suppresses the resync frame too — it is the caller asking for
events only, and watch(ctx, O, stream) sets it. Such a consumer still gets the
recovery, since the re-list is where the new resourceVersion comes from, but not
the state, and has to track expiry itself. Documented rather than papered over.
The fake apiserver's list body is now a function of the request number, so the
test can tell re-listing apart from starting over: p1 exists at the first list
and is gone by the second — exactly the object a replay would never mention.
…enAPI.Clients K8sReflector imports `OpenAPI.Clients: is_longpoll_timeout, is_request_interrupted` to decide whether a dead watch is worth reconnecting. Neither name exists in OpenAPI.jl 1.0, and Kuber had the equivalent judgement internally (k8s_retry_cond, _DECISIVE_HTTP_ERRORS) without exporting it, so a consumer had no supported way to ask. is_retryable(e) is that judgement, named for what it means rather than for what it replaces: "would Kuber retry this", which is the question a consumer driving its own calls has to answer, and it matches the internal vocabulary. It unwraps TaskFailedException and single-exception CompositeException first. That is not incidental — watch(processor, ctx, …) runs the watched call and the processor under @sync, so a failure reaches the caller wrapped, and a predicate that did not unwrap would answer false for every watch failure, the exact case it exists for. A composite carrying more than one exception is left alone: there is no single cause to classify. is_longpoll_timeout has no successor and needs none — watches carry no overall deadline here, so one never ends on a timeout; it ends when the consumer closes the stream, which is not an exception. The C2 entry now carries a migration table for all four lost names, including five `OpenAPI.Clients.ApiException` sites in JuliaRun that the original survey missed.
…o again (C1b) Aggregated APIs and CRD groups are absent from kubernetes/kubernetes release tags because they are not part of Kubernetes: metrics.k8s.io is served by metrics-server, custom.metrics.k8s.io by an adapter, CRD groups by whatever installed them. A live apiserver serves a real OpenAPI 3.0.0 document for each at /openapi/v3/apis/<group>/<version>, so fetch_specs.sh grows a second source mode for exactly that. Provenance goes to SPECS_CAPTURED rather than SPECS_ORIGIN: separate files so neither mode clobbers the other's record, and because a captured document is only as reproducible as the cluster it came from — recorded as server version, context and date. The tag mode also stopped listing files it did not fetch. metrics.k8s.io/v1beta1 is captured (k3s v1.35.4) and shipped. The existing patch rules covered it unchanged, strict generation passed first time, and the other 17 modules regenerated byte-identically. test_metrics covers it live under strict response validation: node metrics, pod metrics in one namespace and across all of them, and single objects of each. It skips with a warning when the cluster has no metrics-server, which includes CI's kind cluster — the group's absence says nothing about whether the client is right. Shipping it *in Kuber* corrects a call OpenAPIv1ConsumerGaps.md made earlier. The 0.2.x line shipped metrics.k8s.io in Kuber — master's SupportedAPIVersions.md lists metrics_v1beta1 and api_typemap.jl has the MetricsV1beta1 aliases — and Metrics.md is a Kuber document, so dropping it was a regression for every Kuber user rather than a JuliaHub-specific gap. The line to draw is whether any user of the API could plausibly have the group: metrics-server yes, an operator's CRDs no. Deployment-specific groups still belong in their own package, plugged in with Kuber.register!.
…need (C5)
On master, list_custom_metrics and list_namespaced_custom_metrics are one-liners
over list(ctx, :MetricValue, "<objecttype>/<name>/<metric>"). The trial replaced
them with an error, because custom.metrics.k8s.io was not generated. They are
those one-liners again, so the group only has to be captured and registered for
JuliaRun's metrics path to work.
Two things had to come back or give way first:
list(ctx, O, name) — the trial's list took no name, so there was nowhere to put
a composite metric name. It is an error to pass one to a list operation with no
path parameter for it, rather than being silently ignored.
_positional no longer insists the path parameter be called `name`. Every group
the apiserver serves calls it that, but emit_registry.jl takes path parameter
names verbatim from the document, and custom.metrics.k8s.io's path is
/namespaces/{namespace}/{compositemetricname} — so the verb layer would have
rejected the very operation the pipeline generated for it. There is only ever
one such parameter, so the name argument fills whichever it is; more than one is
now a clear error instead of a confusing one. This would have bitten any
captured group with a non-standard path, not only this one.
What is not here is the group document. custom.metrics.k8s.io exists only where
an adapter is installed; no public document defines it (master's came from
definitions hand-spliced into the legacy Swagger file) and no cluster reachable
from here serves it. Hand-authoring an OpenAPI 3 document from master's fragment
was considered and rejected — strict response validation would then enforce a
spec nobody could check. The honest version is one --from-cluster capture on a
cluster that has the adapter.
Tests pin what is verifiable without one: the three composite-name shapes, the
helpers reporting an unregistered group rather than erroring outright, the
non-`name` path parameter, and a name refused where the operation takes none.
…h (G13)
Every write in the suite went through the typed 2-arg form, while
services/JobLoops writes through put!(ctx, :Deployment, spec) with a dictionary
rendered from a template. The testset is modelled on hot_standby.jl rather than
invented: the namespace dict is that file's literal shape, and the deployment
comes from JSON.parse of a template.
That choice found something. On JSON.jl 1.x JSON.parse returns a JSON.Object,
which is an AbstractDict but not a Dict — and master's three put! methods are
v::T<:APIModel, v::Dict{String,Any} and v::T<:APIModel, all narrow, so a parsed
template matches none of them and the call is a MethodError. master allows JSON 1
in [compat], so that is a live hazard there rather than a hypothetical. This
branch's v::AbstractDict covers both, and the two cases in the testset are now
deliberately different shapes: a plain Dict for the namespace, a JSON.Object for
the deployment.
Beyond the round trip it pins the things that could break silently: nested
arrays and string maps surviving into the model, the apiVersion coming off the
dict rather than off discovery, and the kind-completion path — a dict with no
"kind", which this branch fills in from the symbol without mutating the caller's
dictionary and without master's unguarded v["kind"] KeyError in exactly the case
the code exists to handle.
Live: 366 assertions against k3s v1.35.4, up from 310.
Found while writing G13's testset: hot_standby.jl scales a deployment with application/json-patch+json and a Vector of operation dictionaries, where the live suite only ever patches with a merge-patch dict. OP_BODIES requires a patch body to be built as the generated Patch type, so whether a bare vector survives that is the first thing G16 should test.
…16, C8)
Enumerating what JuliaRun and JobLoops actually patch — G16's task — turned up a
break rather than a coverage gap. k8s documents ONE request schema for all five
patch media types: meta.v1.Patch, `type: object`. That is untrue of
application/json-patch+json, whose body is an array of RFC 6902 operations, and
the generated Patch model can only hold an object:
DecodeError: expected an object while decoding …Patch, got Vector{Dict{String, Any}}
Which is most of the production patch traffic: julia_parallel_scale (every
worker scale up and down), taint_update_patch, julia_update_job, and
hot_standby.jl's deployment scaling. Only the merge-patch callers worked.
The fix is a patch rule, per the standing rule that a document which lies gets
one. The json-patch content schema becomes an array, declared once as a
component (meta.v1.JSONPatch) and referenced: inlining it per operation makes
the generator emit one item type per patch operation — 132 in apps/v1 alone,
+27 KiB — where the shared component emits one, +3.4 KiB. Items stay untyped
objects, since move/copy carry `from` and remove carries no `value`.
OP_BODIES consequently maps media type -> body type instead of carrying one type
and a list of media types, because a PATCH now genuinely has two body types.
That is a change to the registration contract published in C1a, so register!'s
docstring, its validation and test/register.jl's fixture moved with it.
update! normalizes three caller shapes through _patch_payload: a dict or vector
passes through, JSON text is parsed, and a generated model is encoded to its
JSON object first — JuliaRun patches Secrets with a whole desired Secret, and a
model cannot decode into the open Patch struct. That last one was found by the
live test rather than by reading.
Covered offline in test/simpleapi.jl (per-media body types, the nested-vector
taint shape, and that the object model still refuses an array) and pinned for
every patchable kind in every module by a new test/registry.jl gate, which is
what stops the rule silently not applying to some document. Live: single- and
two-operation json-patches, JSON text, a strategic merge patch, and a Secret
patched with a typed model.
Also recorded, not fixed here: C9, a live defect in JobLoops — networkpolicy.jl
sends a whole policy object under json-patch, which a real apiserver answers 422
— and G12a, that Secret.data now decodes to Vector{UInt8} rather than base64
text, so the natural 0.2.x read idiom decodes twice.
The live testsets create and delete in the same block with no finally, so a failure in the middle leaves objects behind. The next run then fails at put! with a 409 before reaching whatever actually broke — so the second run reports a stale error and hides the real one. That happened twice while working G16: a genuine failure at a two-operation json-patch was followed by a run that aborted at the first put!, and the fix under test was never exercised. ensure_absent deletes an object if present and waits for it to be gone; reset_test_objects runs it over everything a pass creates, before the testsets start. The namespace delete cascades, which covers the objects inside it. CI never saw this because kind is a new cluster every run. It only bites local runs against a long-lived cluster, which is exactly where a failure is being investigated.
Verified on k3s v1.35.4: registry 5664, register 58, helpers 105, simpleapi 90, watch recovery 47, live 472. The live count is up from 366 because the two passes now run to completion rather than aborting mid-testset on leftovers.
K8sReflector watches :Pod with namespace=nothing and a label_selector built by Kuber's own sel helper (k8s_job_pod_monitoring.jl:66). The suite watched a single namespace with no selector, so scope fall-through (:cluster then :allns) and selector handling were covered only as offline resolution. The test mirrors the reflector rather than approximating it, and covers both halves of its loop: the initial get that fills the store, and the watch that maintains it. Two things make it meaningful rather than merely green. A selected pod is created in TWO namespaces, so a result carrying both proves the read is all-namespaces rather than luckily single-namespace. And the watch resumes from the list's resourceVersion, with an unselected pod created BEFORE the selected one — so seeing the selected event proves the other was filtered rather than merely late, where a wait-and-hope negative would be flaky. Found while writing it, and recorded in the doc: put! addresses the request with ctx.namespace and ignores metadata.namespace on the object, so creating an object whose metadata names another namespace is a 400. Not a regression — master does the same and does not even offer the keyword — but it is why the test passes namespace= explicitly.
The recorded 472 was measured on a run that aborted part way through 'Create/Delete Objects' because earlier failures had left objects on the cluster. With ensure_absent/reset_test_objects in place the suite runs to completion at 614.
ReplicaSet, DaemonSet, CronJob, RoleBinding, NetworkPolicy, PersistentVolume and PersistentVolumeClaim, each through the four paths with distinct schemas: create, a get once the controller has written a status, a list with the object still in it, and delete. A get issued straight after put! decodes an empty status block and checks almost none of the kind's schema, so the status poll is the point rather than incidental. Strict response validation checks every kind's schemas independently and two of the six patch rules were found by submitting a kind for the first time, so a seventh rule was the plausible outcome here. None was needed: all seven kinds round-tripped unchanged across four group modules, two of which (rbac.authorization.k8s.io/v1, networking.k8s.io/v1) had never been reached live at all. The fixtures copy the real consumer templates with two deliberate departures. The RoleBinding names a Role that does not exist rather than reproducing JuliaRun's ClusterRole/admin grant -- RBAC permits a dangling roleRef, so the schema is identical and the privilege is not. The NetworkPolicy selects a label no pod carries rather than JobLoops' empty podSelector, which is deny-all- ingress for the namespace: inert under kind's CNI, but it would break the rest of the suite on a cluster whose CNI enforces. Nothing schedules a workload. Also corrects three stale rows in G6's table -- Secret was already covered by G16's patch block, Namespace and ConfigMap by G13's -- and records under C4 that generated field names lowercase the JSON name rather than snake_casing it, which is where this testset spent its debugging. Live suite 614 -> 718.
One ConfigMap carries all three: labels, annotations and data, no controller
writing to it, and creating it makes the ConfigMap list non-empty, which a
list-shape assertion needs in order to mean anything.
G9 rounds labels and annotations through the server. The assertion that earns
its place is the negative one -- metadata.labels is not an AbstractDict and
indexing it the 0.2.x way is a MethodError, which is the exact shape of the
networkpolicy.jl:114 break.
G11 asserts the list shape consumers actually destructure, and turned up G18:
list items are a different Julia type from the standalone object
(IoK8sApiCoreV1PodListItemsItem, not IoK8sApiCoreV1Pod), on every list kind in
every group module. master's PodList.items was Vector{IoK8sApiCoreV1Pod}, so
this is a regression. The k8s document wraps the element $ref in an allOf with
a sibling "default": {}, which makes it a new schema; fields and nested types
are shared, so reads work and nothing had noticed. isa comparisons do not --
JuliaRun api.jl:1418 is one. Fixing it needs a seventh patch rule and a
regeneration, so the test pins the current behaviour and G18 records the
decision.
G10 found that resource_version= never reaches the server on a non-watch read:
the guard "if !watch || resource_version === nothing" computes the result
without it. master has the identical guard in all four verbs, so it is a shared
limitation rather than something the port broke. The test asserts the trap
instead of the intent -- the same impossible version errors as resourceversion=
(the generated spelling, a real query parameter) and succeeds as
resource_version= (the documented spelling, dropped). The fix is split out as
G17 because get's half cannot be fixed by forwarding at all: k8s declares only
"pretty" on read operations.
Also records that the live assertion total moves with cluster activity, since
Watch Events asserts per observed event -- the totals in this file's table are
not comparable across runs.
K8sReflector wraps Kuber.watch in its own `while true` and relies on watch returning on a long-poll timeout so it can re-establish from a resourceVersion it tracked. Watches here carry no deadline and the pump re-watches internally, so a clean server close never ends the watch -- including one caused by timeoutseconds, which means there is no server-side way to hand control back either. The reflector's loop is dead code on this branch regardless. What the test establishes is that the pattern is still expressible: a stream processor that leaves its event loop closes the stream through the finally in watch(streamprocessor, ...), the watch ends, and the caller re-enters with its own resource_version. Two rounds against the fake apiserver, asserting the second watch query carries the version the first round's last event reported. The assertion worth having is the last one: the resumed round issues no list request. A caller supplying resource_version skips the initial list, so a consumer keeping its own store pays for full state exactly once -- the useful half of the same `if !watch || resource_version === nothing` guard that makes G17 a bug on non-watch reads. Ticked with the distinction stated in the doc, since "G2 done" would otherwise read as "the reflector is fine": the box that remains open is porting its loop. watch_recovery.jl 47 -> 56.
The other re-watch tests assert that a resume happens with the right resourceVersion. This asserts the property a cache-maintaining consumer actually depends on: the sequence either side of the seam is exactly what the server sent. Two things keep it from restating the resume assertion. The first watch sends a burst of three events and then closes cleanly, so events are in flight when the connection ends rather than arriving one per round trip; and the consumer reads nothing until the seam has demonstrably passed -- it waits on the second watch request appearing, not on an event -- so the burst has to survive buffered across the re-watch. Assertions: the exact sequence, allunique, an empty stream afterwards (a re-delivered frame would be sitting in it), and that the resume names the last version of the burst rather than the first. Documented alongside what it does not prove: that no duplicate arrives, only that Kuber does not manufacture one. Kuber cannot deduplicate -- watch events carry no identity beyond the object and its version -- so a consumer needing exactly-once must key on metadata.resourceversion itself, as it would with client-go. Worth stating because "continuity is tested" invites the stronger reading. This closes the watch-contract cluster (G1-G4) apart from the reflector port under G2 and the deferred G5. watch_recovery.jl 56 -> 70.
JuliaRun builds Secrets whose data values are raw Vector{UInt8}:
_as_binary_secret (api.jl:203-214) base64-decodes anything that looks base64
before handing it over, so what reaches Kuber is always bytes. The 0.2.x client
base64-encoded them onto the wire because the field is format: byte, and the
1.0 runtime does the same in both directions, so those values survive the port
unchanged. The container does not -- data is an open struct now, so
Secret(; data=bindata) becomes
Secret(; data=SecretData(additional_properties=bindata)).
The test writes bytes that are deliberately not valid UTF-8, so a round trip
that "works" by treating the value as text cannot pass it.
Two behaviours pinned because they are easy to guess wrong:
- stringData is write-only. The apiserver folds it into data and never returns
it, so a consumer that writes it must not expect to read it back.
- A merge patch merges the map rather than replacing it (RFC 7386): patching
data with only "token" leaves "binary" and "plain" intact, and only an
explicit null removes a key. This was asserted the wrong way round first and
the live run corrected it. update_secret sends the whole desired map, so it
is unaffected either way -- but a caller who believed the map were replaced
when it merges would be safe, and one who believed the reverse would silently
drop every other secret in the object.
…e (G8) The Namespace half of G8 was already covered by create_delete_from_dicts (G13) and PersistentVolume by create_delete_more_kinds (G6). Node was left, and Node is different in kind: no consumer creates one. The monorepo's set_node_label, set_node_cordon and taint_update_patch all patch an existing node. Creating a Node object through the API works, but it would test an operation nobody performs and leave a kubelet-less NotReady node on the cluster for metrics-server and the scheduler to trip over -- so this patches a real node and puts it back. Both consumer shapes: - a merge patch setting a label, then an explicit null removing it, which pins the complement of what G7 found: unmentioned keys survive a merge patch and null is the only way to delete one. - a json-patch whose value is a nested array of dicts (taint_update_patch's shape), appending via /spec/taints/- rather than replacing /spec/taints so the node's existing taints are untouched. A control-plane node carries one, and dropping it would be a live change to the cluster rather than a test. The taint is PreferNoSchedule and nothing cordons, because the rest of the live suite schedules pods on that same node. Reading a Node by name also exercises the :cluster scope fallback for a kind that is neither Namespace nor PersistentVolume.
characterize_retries.jl pins the exception types the runtime raises, is not in runtests.jl, and never drives the retry loop. test/retries.jl drives it: a server that always fails with a chosen status and counts requests, so "retried" is the difference between one request and several rather than something inferred from timing. Offline and deterministic -- provoking real load-shedding from an apiserver is neither reliable nor cheap, and the interesting variable is the status, not the cluster. With HTTP.jl's own retry layer switched off so the count reflects Kuber's loop alone, at max_tries=3: 500/502/503/504 take 4 requests and are retryable; 404/409/422 and 429 take 1 and are not. The watch establish call -- the only thing in the watch path k8s_retry wraps -- retries and then surfaces. Writing it turned up two things, both shared with master and both recorded as decisions rather than changed here: G19: 429 is not retried. Kubernetes sheds load with 429 plus Retry-After and client-go retries it; k8s_retryable_codes omits it on both branches, so a throttled call fails at once where client-go would have absorbed it. G20: HTTP.jl 2.x retries idempotent requests underneath Kuber, five requests per Kuber attempt at this pin -- max_tries=1 is ten requests, not two, and set_retries(count=5) is thirty rather than six, with both backoffs composing. max_tries is also off by one: it is a retry count, so max_tries=1 means two attempts, and mutating calls take retries(ctx, true) == 1, which means a put! whose first attempt 5xxs is retried once. That is not what "only non-mutating calls retry by default" implies. The two interact -- fixing G19 means little while HTTP.jl retries 429 anyway, outside max_tries and without honouring Retry-After -- so they are grouped in the suggested order.
…(G18, G17)
Patch rule 7. k8s never writes a bare $ref for a property: it wraps it in a
single-element allOf so it can hang a description beside it, because a $ref
with siblings is undefined in OAS 3.0. Read literally that wrapper is a new
schema, so the generator minted a type per use site:
Pod.spec was IoK8sApiCoreV1PodSpec2 -- the 2 disambiguating it from
the real PodSpec component, which nothing referenced
Pod.metadata was IoK8sApiCoreV1PodMetadata, not the shared ObjectMeta
PodList.items was Vector{IoK8sApiCoreV1PodListItemsItem}, so
`item isa kind_to_type(ctx, :Pod)` was false
The last of those is G18, a regression from master where all three were one
type. JuliaRun/src/kubernetes/api.jl:1418 compares types on an object that came
out of a list.
1290 sites across the 18 documents, in exactly two positions: property schemas
and the items of array properties. Generated types 2252 -> 1098, tree 24 -> 18 MB.
Two things about the rule:
- Scoped to those two positions rather than walked recursively. apiextensions'
JSONSchemaProps describes JSON Schema itself, so it has properties *named*
allOf, nullable and items; a recursive walk rewrites that map and silently
corrupts the CRD document. Verified it survives intact.
- Guarded on shape -- single-element allOf whose element is a bare $ref --
rather than on the survey that says every allOf looks like that. A future
spec bump introducing a two-element allOf is left alone to be noticed.
test/registry.jl gates it on type identity across four kinds in three group
modules, plus no ListItemsItem name surviving anywhere. A collapse that
produced an alias per use site would shrink the diff by as much and still be
wrong.
Also G17's list half: list now forwards resource_version to the operation's
resourceversion parameter on the non-watch path. Inside a watch it still means
"resume from here" and is consumed by the pump, which is a different thing.
G17's get half is deliberately NOT done. The apiserver does honour
resourceVersion on a single read -- verified live, an impossible version answers
504 -- so a patch rule would be defensible. It is held back because it would
change which requests Kuber can construct, and its natural failure is a 504 that
blocks for the apiserver's wait, multiplied by a retry budget that G20 shows is
not what it appears (max_tries=1 is ten requests). Cheap to add once G19/G20 are
settled, and pointless before.
Strict generation and strict response validation stayed on. Full suite green
against a live cluster: registry 5694, retries 47, watch recovery 70, live 756.
The performance table in OpenAPIv1TrialResults.md predates this and is marked
stale rather than re-quoted -- half the types can only have improved
precompilation and TTFX, but by how much is unmeasured.
…budget G19 and G20, done together because neither makes sense alone. G20. HTTP.jl 2.x retries idempotent requests on a retryable status by default, underneath k8s_retry -- five requests per Kuber attempt at this pin. So max_tries bounded nothing (max_tries=1 was ten requests, set_retries(count=5) was thirty), both backoffs composed, and a mutating call could be retried by a layer with no notion of which calls mutate. Kuber already had a curated policy -- a status list, a mutating-vs-not rule, is_retryable as its public face -- and HTTP.jl's layer silently contradicted all three. _call_options now merges retry=false into every call, so Kuber's loop is the only one; set_request_options(ctx; retry=true) hands it back. It goes there rather than on the context because client_kwargs are passed to the generated Client constructor, which takes no retry -- putting it there is a MethodError on the first call. k8s_delay clamps to max(0, max_tries - 1) delays, so max_tries counts attempts. Two visible consequences: default_retries=5 is 5 attempts rather than 6, and a mutating call is 1 attempt rather than 2 -- the contract set_retries(all_apis=false) always claimed, and the direction that avoids a duplicate create. G19. Kubernetes sheds load with 429 plus Retry-After and client-go retries it; k8s_retryable_codes omitted it, so a throttled call failed at once. Before G20 HTTP.jl retried it anyway, making the observable behaviour "retried, but not by Kuber, not honouring Retry-After, and not counted by max_tries" -- worse than either answer. With HTTP.jl's layer off, Kuber's list is the whole story, so 429 joins it. Retry-After is a floor on the backoff: it only ever lengthens a wait. Scoped to 429 (a 5xx carrying the header should not reprice every retry in the client), capped at 30s, delta-seconds only -- the HTTP-date form parses to nothing and falls back to the backoff, which is the safe direction. Honouring it is why k8s_retry is an explicit loop now: Base.retry takes its delays from an iterator that never sees the exception. Docstrings that would otherwise have become lies are updated rather than left: set_retries' "how many times to retry", k8s_retry's "if max_tries > 1", set_request_options, and README's retry bullet. characterize_retries.jl is not in runtests.jl and still prints correctly, but now notes that the loop beneath it changed. This unblocks G17's get half, whose objection was a slow 504 multiplied by an unbounded retry budget. Full suite green: registry 5694, register! 58, helpers 106, simpleapi 90, retries 56, watch recovery 70, live 744.
Patch rule 8, and the last of G17.
k8s documents resourceVersion on every list operation and on none of the reads,
but the apiserver honours it on both -- verified live: GET .../configmaps/x?
resourceVersion=0 answers 200, and a version the cluster has never reached
answers 504 "Too large resource version". So a consumer asking for a read "not
older than" a version it already saw had no parameter to send. K8sReflector
does exactly that (K8sReflector.jl:136-141), and that call had never done what
it reads as, on either branch.
Added only to paths ending in {name}: the object read itself, not subresources
like pods/log where a resource version is meaningless. 31 operations in core v1,
none of which declared it already. The parameter goes at operation level; the
path-level name/namespace/pretty are untouched.
`get` forwards resource_version on the non-watch path, as `list` already does.
Inside a watch it still means where to resume from and is consumed by the pump.
Sequenced deliberately after G19/G20. Rules 1-7 make the document describe what
the server already does with requests Kuber already sends; this is the first
that changes which requests Kuber can construct, and its natural failure is a
504 that blocks for the apiserver's wait. That was not worth adding while
max_tries=1 still meant ten requests.
The live assertion is that an impossible version now answers 504 through `get`
-- which is what proves the rule reached the wire rather than only the document
-- plus the reflector's actual shape: read, keep the version, read again not
older than it.
Full suite green: registry 5694, register! 58, helpers 106, simpleapi 90,
retries 56, watch recovery 70, live 752.
Patch rules 7 and 8 halved the generated type count (2252 -> 1098), which made the numbers in OpenAPIv1TrialResults.md section 2 stale in the branch's favour. Re-measured on the same cluster and methodology, kept alongside the originals because the delta is the point: precompilation ~22 s -> 14.4 s using Kuber 0.36 s -> 0.24 s discovery 0.22 s -> 0.22 s (unchanged, as I/O should be) TTFX first list ~15-16 s -> 12.4 s steady list (2 pods) 11.5 ms -> 8.6 ms validation share 78 % -> 72 % second group module 1.2 s -> 1.0-1.3 s watch reaction 5.6-11.2ms -> 5.5-10.4 ms, 0 missed registry emission ~100 s -> 95.4 s Everything that is compilation got about a third cheaper; nothing that is I/O moved. TTFX only came down ~3 s, which says the remainder is the validation engine and the operation rather than the model types -- a PrecompileTools workload is what would move it. The steady-state row nearly produced a false result. The first measurement said 116 ms, a tenfold regression, and the cause was the cluster: `default` held 35 pods and the call is dominated by per-item response validation. The original 11.5 ms was taken at 2 pods, so that condition was recreated. The doc now says the number is meaningless without its item count. Chasing those 35 pods found the leak that produced them: delete!(ctx, :Job, ...) sends no propagation policy, so the apiserver orphans the Job's pods -- about two per run. CI never sees it because kind is fresh each time; a local or shared cluster accumulates them, and they are not inert, as the 116 ms shows. propagation_policy="Background" on the Job delete and in ensure_absent. Verified: the run after this fix left zero pods in `default`, where it would have left two. Rerunning the whole generation chain also reproduced src/ApiImpl/generated/ and the patched specs byte-identically to what is committed -- the reproducibility claim gen/openapi_v1/README.md makes, checked rather than asserted. Full suite green: registry 5694, register! 58, helpers 106, simpleapi 90, retries 56, watch recovery 70, live 760.
…sumers (G12, G12a)
The Kuber half of G12 is one testset: a pod carrying both limits and requests,
read back through kuber_props, with Quantity's shape pinned (still a struct with
a single value field, so JuliaRun's string(cpu.value) survives). It also asserts
the negative -- neither map is a dictionary, so `in keys(res)` and `res["cpu"]`
are MethodErrors rather than wrong answers -- and that after patch rule 7
`resources` is the shared ResourceRequirements rather than a positional copy per
container-bearing kind.
Both audits produced consumer fixes, not Kuber fixes.
G12a found one real site. JuliaRunPool.jl:135-136 compares a value read back
from a Secret against the base64-encoded config string. On 0.2.x Secret.data came
back as base64 text and that was correct; here it is a Vector{UInt8} of
plaintext, so the comparison can never hold. The branch exists to skip a
redundant write, so nothing errors -- the image-pull secret is rewritten on every
namespace creation instead. Everything else is clear: JuliaRun's base64decode
sites are kubeconfig parsing, env packing, or the write side, and the monorepo's
kill_k8s/monitoring_loop blobs come from the database, not from a Secret.
G12 found two breaks in clustermgmt.jl. keys(res)/res["cpu"] on an open struct is
the loud one. The quiet one is hasproperty: on 0.2.x an unset field was missing,
on 1.0 every field exists and unset is ABSENT, so hasproperty(resources,
:requests) is always true and container_resource returns ABSENT for a container
declaring only limits -- never reaching its elseif, and the caller's `res ===
nothing` guard does not catch it.
That pattern surfaced the largest unrecorded item in the document, now added to
C2: JuliaRun imports getpropertyat/haspropertyat from OpenAPI.Clients across 49
call sites, and they do not exist in 1.0. Recorded with a recommendation to
export Kuber equivalents built on _field, treating ABSENT and nothing alike --
not implemented, because a new public API is not a routine call. It also moves 49
sites from "rewrite" to "reimport", which changes how C1c should be costed.
Full suite green: registry 5694, register! 58, helpers 106, simpleapi 90,
retries 56, watch recovery 70, live 812.
OpenAPI.jl 1.0 dropped OpenAPI.Clients.getpropertyat/haspropertyat, and JuliaRun
imports them at 49 call sites -- the largest single porting item in
OpenAPIv1ConsumerGaps.md, missed until the G12 audit. These are the
replacements, in src/helpers.jl.
Kept unexported on purpose: they are a shim for consumers porting off 0.2.x
rather than a shape this API wants to encourage, so a call site has to write
Kuber. and stays easy to grep for later. For JuliaRun the change is the import
line, not the 49 uses.
Three decisions:
ABSENT counts as absent. This is the whole reason they exist. 0.x overrode
Base.hasproperty on models to mean "not nothing", so haspropertyat asked a real
question; on 1.0 every field exists, so a handwritten walk answers true for
everything -- the trap that makes JuliaRun's container_resource return ABSENT
instead of falling through to limits. The test pins both sides:
hasproperty(pod, :status) is true while haspropertyat(pod, :status) is false.
A path element may name an open-struct entry, so
getpropertyat(node, :metadata, :labels, "role") reads a label without a separate
kuber_props call. That covers the get(nodelabels, "role", "") sites in
clustermgmt.jl:203-204 as well.
Case is not folded. JuliaRun's calls include :loadBalancer, :nodeName and
:backoffLimit, which are C4 breaks at the same sites. Folding would have fixed
those silently, at the cost of a genuine typo succeeding whenever it happened to
lowercase-match. A wrong name reads as absent, and the test asserts both
directions.
0.x's odd behaviour of returning a Vector{Bool} when a vector is met by a
non-integer path element is kept rather than quietly improved -- the job of a
shim is to be a drop-in.
Full suite green: registry 5694, register! 58, helpers 130, simpleapi 90,
retries 56, watch recovery 70, live 816.
C1b drew the boundary for captured groups -- a group belongs in Kuber when any user of the API could plausibly have it -- and moved metrics.k8s.io in on it. The same argument reaches custom.metrics.k8s.io and the documents never followed: seven passages still said it belongs in JuliaHubK8sApi or "the deployment's own package", which contradicted C1b in the same file. It belongs here. The helpers that call it are exported from Kuber and Metrics.md documents them at length; 0.2.x shipped the group in Kuber too; and the schema is adapter-independent boilerplate from custom-metrics-apiserver, so what varies per adapter is metric names, which are path parameters rather than types. Shipping the helpers here and the group they call elsewhere is the incoherence C1b just corrected once. Refines the rule while it is being applied: the test is now prevalence *and* whether the schema varies with the deployment. Operator CRDs fail the second and stay out; custom metrics passes both. That last claim is the load-bearing one, so C5 now says to check it at capture time rather than assert it -- components.schemas holding only MetricValue, MetricValueList and shared meta, and the path parameter really being called compositemetricname, which _positional was fixed for against master's hand-spliced fragment and never against a served document. A metric name in the schema would flip the decision back. Also records that the capture is more reachable than "a cluster this repo cannot reach": any conformant adapter serves the same document, so the custom-metrics-apiserver test-adapter, which needs no Prometheus behind it, is enough. No code changes. README's bullet was stale in a second way and is fixed with the rest: it still said the custom-metrics helpers throw and that metrics.k8s.io was out of trial scope, both untrue since the 2026-08-14 capture.
C5 said what was needed -- one capture of custom.metrics.k8s.io/v1beta1 -- and left "on a cluster running an adapter" as the whole instruction. Looking for a recipe turned up two things that change the shape of the step. JuliaRun already ships prometheus-adapter manifests with custom metrics (scripts/local/compute/metrics/prometheus/manifests/), which looks like the obvious source and is not one: the image is quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0, from 2019 and built long before aggregated API servers served /openapi/v3 at all, and the APIService is apiregistration.k8s.io/v1beta1, which k8s 1.22 removed. C5 now says so, because the failure mode is an afternoon spent on a manifest set that cannot produce the document. That is also what makes the gate worth stating first. fetch_specs.sh --from-cluster does one thing, kubectl get --raw /openapi/v3/apis/<gv>, so whether a cluster can produce the capture is answerable in one command before anything is deployed -- and the answer is a property of the adapter's age, not of the command. Verified the jq shape against the local k3s, where it prints apis/metrics.k8s.io/v1beta1. Three routes recorded in provenance order: a JuliaHub cluster (the document production actually serves, blocked on access), a current prometheus-adapter on the local k3s, and the custom-metrics-apiserver test-adapter, which serves static values with no Prometheus behind it -- test-adapter-deploy/testing-adapter.yaml, no published image, and a k3s wrinkle since make test-kind loads into kind. Second finding, smaller: those same manifests register v1beta1 only, with no v1beta2 APIService. That is evidence on C5's second checkbox rather than an answer to it -- a dev-deploy path is not production -- so the box stays open with the expected answer recorded.
Stood up prometheus-adapter v0.12.0 on the local k3s with no Prometheus behind
it, captured the served OpenAPI v3 document, and took the adapter down again.
The document is kept under gen/openapi_v1/reference-captures/ -- deliberately not
in specs/, which the patch and generate steps glob.
The schemas are exactly what C5 predicted: MetricValue, MetricValueList and
shared meta types, no metric name anywhere. Metric names are discovered at
runtime, which is why the resource list came back empty with no Prometheus while
the document was complete. So the adapter-independence claim holds, and testing
it was still not enough -- the operations are what decide the cost, and they do
not fit:
- No operation carries x-kubernetes-group-version-kind. emit_registry.jl skips
those, so OPS would get nothing while KIND_TYPES populated normally from the
schemas, which carry GVK. kind_to_type(:MetricValue) would work and
list(ctx, :MetricValue, ...) would not resolve -- worse than not shipping.
- Metrics are addressed through {resource}/{name}/{subresource}: three path
parameters. _positional takes one, and the emitter's plural-to-kind pass keys
on a literal path segment that a variable cannot supply.
Against that: nothing calls the API. The earlier audit counted references, not
callers. JuliaRun's KubernetesMetrics module is the only code touching custom
metrics *or* metrics.k8s.io, nothing constructs a KubernetesMetricsCtx except a
standalone probe that runtests.jl never includes, and the monorepo has no
reference to either group in any Julia file. metrics.k8s.io stays regardless --
already shipped and live-tested, near-universal, and 0.2.x had it.
Corrects a factual error the capture exposed, in src/simpleapi.jl and in C5:
custom.metrics.k8s.io does not call its path parameter compositemetricname.
That was master's hand-spliced Swagger collapsing three parameters into one, a
fiction that produced the right URL because joining them with / is the composite
string. The _positional relaxation stays -- accepting some other single name is
still right for captured groups -- but it was never what this group needed.
Also records a trap worth knowing for any capture: /openapi/v3 listed v1beta2 as
well, and v1beta2 is not reachable -- no APIService registers it, /apis/... 404s,
and discovery offers v1beta1 alone. An entry in /openapi/v3 means the aggregated
server compiled the version in, not that the cluster serves it. That answers C5's
v1beta2 checkbox too.
The cluster-capture mode wrote its provenance with a plain `>` and listed only the files of the run that wrote it. So capturing a second group destroyed the first group's record -- in the one file whose entire job is to remember where a document came from, and silently, since the surviving record looks perfectly well-formed. Capturing custom.metrics.k8s.io this way would have erased how metrics.k8s.io was obtained. SPECS_CAPTURED is now one record per file -- file, group version path, server, context, date, checksum -- and a capture replaces the records for the files it writes while leaving the others alone. Groups captured months apart from different clusters each keep their own provenance, which the old single header block could not express anyway: it stamped one server and one date across whatever happened to be listed. Two smaller things in the same path: - A capture went through `kubectl ... | jq . > "$out"`, so the redirect emptied the existing document before kubectl was known to have failed. It writes to a temp file and moves on success, and says which document it left alone. - SPECS_DIR overrides the destination directory, so the capture path can be run against a throwaway directory instead of the checked-in specs. The checked-in record for metrics.k8s.io is migrated to the new format by hand, and verified byte-identical to what a fresh capture now produces. Exercised against the local k3s: re-capturing a group replaces its own record, capturing a different group leaves the first intact (the bug), two groups in one invocation give two records, and a failed capture leaves both the document and the provenance file untouched with no temp files behind.
The merge identified a record by its file: line and skipped anything without one, so a hand-edit that split a record -- a stray blank line in the middle -- lost the half that did not carry the name, silently. That is the same failure the merge was written to prevent, just at a smaller scale, and hand-editing is not hypothetical: the existing record was migrated to this format by hand. Unrecognized blocks are now kept verbatim with a line to stderr. A mangled file stays mangled and visible rather than quietly getting shorter. Also states C5's emitter claim as what it is. The captured operations carry neither x-kubernetes-group-version-kind nor x-kubernetes-action, and ops() needs both -- the first pass to learn a resource's kind, the second to drop anything it did not learn and anything without an action verb. That is read off the document and emit_registry.jl:245-262 rather than measured by running the chain, and the document now says so.
G5 asked whether a long-lived watch is testable in CI and had stood deferred on "probably not". The premise was wrong: what makes a watch long-lived is the apiserver's own timer -- min-request-timeout defaults to 1800s and the handler randomizes within [1800, 3600) -- so an unbounded watch is closed at 30 to 60 minutes, and guaranteeing one close means an hour per matrix entry. But the hour of waiting is not the mechanism. The close is, and a close is producible in seconds: timeoutseconds on the request ends a watch with exactly the same clean close (verified against a live cluster), and min-request-timeout is settable on a kind cluster through kubeadmConfigPatches. So the item splits: - G5a, CI-sized: a server-ended watch re-establishes and loses nothing, allowwatchbookmarks produces BOOKMARK events that do not disturb the stream or the tracked resourceVersion, and an expired resourceVersion resyncs. The first and third are widenings -- watch_recovery.jl covers them against the fake apiserver, and what is untested is that a real one behaves the same. BOOKMARK is not: nothing exercises it anywhere, Kuber never asks for bookmarks, and a bookmark carries an object holding only a resourceVersion, a shape that has never been through KuberEvent. That leg is where this is most likely to find something. - G5b, manual probe: descriptor and memory growth over hours, a load balancer dropping an idle connection, an apiserver rollout mid-watch, HTTP/2 GOAWAY. Real wall time or real infrastructure, with watch_latency.jl as the precedent for how such a probe is carried. The LB idle timeout is worth calling out -- 60 to 350 seconds on the common cloud balancers, the failure JuliaRun is most likely to meet in production, and the one a kind cluster cannot produce at all because there is no intermediary in front of it.
long_lived_watch in test/runtests.jl, about twelve seconds of wall time for what G5 had recorded as an hour-long test nobody could run in CI. timeoutseconds on the request ends a watch with exactly the clean close the apiserver's own timer produces, so three server-initiated closes cost ten seconds and exercise the same path. The kind kubeadmConfigPatches route for min-request-timeout turned out not to be needed at all. Three legs, all of them already covered against watch_recovery.jl's fake apiserver -- which is the point, since what was untested is that a real server ends a watch the way the fake does: - pods created either side of a server-initiated close each arrive exactly once, so nothing is dropped across the gap and nothing replayed after it - BOOKMARK events arrive when asked for and disturb neither the stream nor the tracked resourceVersion. Nothing exercised these anywhere before: Kuber never sets allowwatchbookmarks, and runtests.jl only asserted a bookmark would be tolerated if one turned up - resuming from resourceVersion=1 provokes the in-stream ERROR/410 immediately, and the resync frame that follows carries complete current state Nothing broke, and one rule got confirmed on a payload it had never seen. A bookmark's object is the watched kind carrying metadata.resourceVersion and, for a Pod, spec.containers as an explicit null -- a required array property. It decodes only because patch rule 2 makes array properties nullable, the Go-nil-slice rule, which until now was exercised against list and read payloads rather than watch frames. The test asserts that null rather than stepping over it, so a regression in the rule fails here too. Live suite 816 -> 883 assertions; full suite green.
The new testset passed locally and failed on kind: one pod's ADDED never arrived, and only the first of the three, which is the shape of a race rather than a broken watch. watch(ctx, O, stream) takes no resource_version, so it lists internally to learn where to resume and discards that list. An object created between the caller's watch call and that internal list is inside the list, is thrown away with it, and is never announced. Locally the create lost that race; on a slower cluster it won. The test now seeds the version from a get of its own -- the same list-then-watch shape K8sReflector and watch_selector_all_namespaces already use -- which closes the window rather than widening a timeout around it. Not a defect: the form is documented as events-only. But "no initial state" and "a silent hole at establish time" are different promises and only the first was written down, so README says it now, next to the events-only form, with the remedy. Live suite 898 assertions, full suite green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A trial rebuild of Kuber.jl on OpenAPI.jl 1.0 (JuliaComputing/OpenAPI.jl#103), replacing the entire generated API layer and rewriting the verb layer against the new runtime. It follows
OpenAPIv1TrialBranchPlan.md;OpenAPIv1RewriteNotes.mdis the evaluation behind it.Read
OpenAPIv1TrialResults.mdfirst. It is the implementation record: what was built, the five places implementation contradicted the plan and why, the measured numbers, and what is left. Where the plan and the results doc disagree, the results doc is current.Draft, and it cannot merge as-is:
Project.tomlpins OpenAPI via[sources]to an unmerged commit (quinnj/OpenAPI.jl@1ff9ba8). Generated output is byte-stable only within a pinned commit, so when JuliaComputing/OpenAPI.jl#103 is merged and tagged the pin has to be dropped and everything regenerated.Shape of the change
src/ApiImpl/generated/— one generated module per Kubernetes group version (K8sV1,K8sAppsV1, …), 17 group versions, each carrying its own models, operations and embedded JSON Schemas. Replaces the oldsrc/ApiImpl/api/tree.src/ApiImpl/generated/registry.jl(also generated) — the lookup tables the verb layer resolves through:GROUP_MODULES,MODULE_GVS,KIND_TYPES,OPS,OP_PARAMS,OP_BODIES. These replaceapi_typemap.jl/api_versions.jland all the string-munging plusevallookups.src/helpers.jl,src/simpleapi.jl— rewritten.KuberContextnow holds oneRuntime.Clientper group module (a client is bound to its module's compiled spec and cannot be shared), HTTP.jl 2.x request options, and a retry condition built from an actual characterization of the runtime's exception types rather than guesswork (test/characterize_retries.jlrecords the findings).gen/openapi_v1/— the reproducible generation pipeline: fetch pristine OpenAPI v3 documents from akubernetes/kubernetesrelease tag, patch them withpatch_k8s_spec.jq, generate in strict mode, emit the registry. See its README.get,list,put!,update!,delete!,watch,sel) is unchanged in shape.Strict generation and strict response validation are on, throughout. A
SchemaValidationErroragainst a real cluster means the spec lies and the fix is a new patch rule — there is novalidate_responses=falseanywhere insrc/. Two of the five patch rules were found exactly that way.What changes for callers
ABSENT, notnothing— the one semantic change to watch for.nothingnow means an explicit JSONnull. Code testing "not set" withx.field === nothingmust useKuber._field(x.field) === nothing._suffix on collisions:metadata.resourceversion,obj.apiversion,event.type. Type names are unchanged (IoK8sApiCoreV1Pod).Dicts.metadata.labels/annotationsentries live inadditional_properties; usekuber_props(pod.metadata.annotations)["key"].KuberEventwith an already-typedevent.object; thekuber_objround-trip is gone.apps/v1'sStatusis not core's. Comparekuber_kind(res) == "Status", not the type — this matters fordelete!.set_timeoutsetsrequest_timeout; watches deliberately carry no overall deadline (bound them withtimeout_seconds).KuberException— no(result, response)tuples.Watches
The subtlest part, and worth reviewing closely. There are no dedicated watch operations — the deprecated
/watch/paths are deliberately not patched back in. Watching iswatch=trueon the list op with an accept-scoped codec (application/json;stream=watch), which fires only for calls that ask for it, because a real apiserver replies bareapplication/json. Since the generated call returns at the response head,list's watch branch pumps the stream inline and returns only when the watch is over — that is what keepswatch(processor, ctx, list, O)and itsfinally close(stream)blocks working.Four watch-lifecycle bugs were found and fixed during the trial (consumer-close as the only stop signal, resume from the last
resourceVersion, in-stream 410 restarting without one, and a re-watch spin on a 200-with-no-events).test/watch_recovery.jlcovers all of it against a fake apiserver.Testing
CI is green on Julia 1.11, 1 and nightly (run 31702176819) — 3824 offline assertions plus a live suite against a Kubernetes 1.35
kindcluster.test/registry.jltest/helpers.jltest/simpleapi.jltest/watch_recovery.jltest/runtests.jllive suiteregistry.jlis the generation gate: every table entry resolves and no deprecatedwatch*operation leaks in. The live suite is skipped with a warning when no server is reachable.Note that CI's cluster version is now load-bearing, which it was not on the 0.2.x line: responses are validated against the v1.35.4 schemas the client was generated from, so the workflow pins kind
v0.32.0with thev1.35.5node image. A spec bump has to move that pin with it.Not in
runtests.jl, kept as manual probes:test/characterize_retries.jl(pins the runtime's exception types) andtest/watch_latency.jl(measured 5.6–11.2 ms reactions, 0 missed events).Known limitations
Out of trial scope by decision:
Metrics.mdis annotated.custom.metrics.k8s.ioneeds a document captured from a cluster that serves it.metrics.k8s.io) and CRD groups likewise — neither appears in upstream release-tag specs.Found during the trial:
listis ~15 s. APrecompileToolsworkload is the obvious next step.Before this can come out of draft
[sources], set compat to the tag, regenerate everything.=== nothingon model fields and formetadata.labels/annotationsindexing.kubectl proxy, so the credential path is untested against real TLS and bearer tokens.Two things for reviewers to rule on: whether
gen/openapi_v1_prototype/(5 files of reference code from the evaluation) belongs in the repo or should be dropped before merge, and whether the three trial documents should be consolidated once this graduates.