diff --git a/.agents/skills/effect/SKILL.md b/.agents/skills/effect/SKILL.md new file mode 100644 index 0000000000..1027916b35 --- /dev/null +++ b/.agents/skills/effect/SKILL.md @@ -0,0 +1,98 @@ +--- +name: effect +description: | + Opinionated guide for building production TypeScript applications with Effect v4. Use when implementing Effect workflows, services, layers, schemas, configuration, schedules, caches, streams, HTTP clients, or tests. +license: MIT +compatibility: Requires Effect v4. Examples are reviewed against the version documented in this repository. +--- + +# Effect + +Use current Effect v4 APIs and the production defaults in this skill. Established project conventions still take precedence unless the task is explicitly changing them. + +## Source Rule + +Check these before guessing: + +- the nearest `AGENTS.md` and any project-local Effect practices doc +- the project-pinned `effect` package source and version +- current upstream Effect source when the installed package does not answer the question + +## Branch Chooser + +Read only the branch references that match the task. + +- Data models, schemas, brands, variants, optional keys, or decoders: read `references/SCHEMA.md`. +- Services, module surfaces, layers, runtime wiring, errors, `Effect.fn`, or test services: read `references/SERVICES_LAYERS.md`. +- Runtime config, env variables, `ConfigProvider`, or `layerConfig`: read `references/CONFIG.md`. +- Retry, repeat, polling, backoff, jitter, rate-limit-aware policies, or pass loops: read `references/SCHEDULING.md`. +- Memoization, per-key TTL caches, deduplicating concurrent lookups, or request batching: read `references/CACHING.md`. +- Streams, event sources, async iterables, queues/pubsubs, pagination, backpressure, or stream consumers: read `references/STREAMS.md`. +- Outgoing HTTP calls, Effect HttpClient, status handling, or HTTP rate limiting: read `references/HTTP_CLIENTS.md`. +- Effect tests, time, sleeps, concurrency synchronization, or fakes: read `references/TESTING.md`. + +If a task spans several branches, read all matching files before editing. + +## Core Defaults + +- Compose workflows with `Effect.gen(function* () { ... })`. +- Define public service methods and non-trivial internal service methods with `Effect.fn("Domain.operation")`. +- Use `Effect.fnUntraced` only for internal helpers where stack-frame/span metadata is intentionally unnecessary. +- Prefer `Context.Service` for application services when the codebase has not standardized on another current service-tag style. +- Build real service implementations with `Layer.effect(Service, Effect.gen(...))` and return `Service.of({ ... })`. +- Model records with `Schema.Struct(...)` plus a same-name `interface`. +- Model typed Effect errors with `Schema.TaggedError`. +- Read runtime config through `Config`, not direct `process.env` access in application logic. +- Use `Schedule` for retry, repeat, polling, pacing, and backoff policies. +- Use `Stream` for effectful sources that emit many values over time and need pull, backpressure, interruption, or transformation. +- Prefer Effect HTTP client modules for outgoing HTTP in Effect applications when their typed errors, layers, and client transforms are useful. +- Prefer Effect-aware tests, explicit layers, and deterministic synchronization over sleeps. +- Prefer decoders and `schema.makeEffect(...)` at untrusted boundaries; reserve throwing `schema.make(...)` for trusted construction, and never use casts to skip validation. + +## Quick Selection Guide + +- Ordinary object record: `Schema.Struct(...)` plus same-name `interface`. +- Scalar ID/value object: constrained branded schema. +- Internal workflow decision or state: `Data.TaggedEnum<...>` plus `Data.taggedEnum<...>()` constructors and exhaustive `$match`. +- Reusable boundary-crossing tagged variant: `Schema.TaggedStruct(...)` plus same-name `interface`. +- Boundary-crossing tagged union: `Schema.TaggedUnion(...)` with `.cases`, `.guards`, and `.match`. +- External/custom discriminator such as `type`: `Schema.Struct({ type: Schema.tag("variant"), ... })` plus `Schema.toTaggedUnion("type")` when union helpers are needed. +- Expected typed failure: `Schema.TaggedError`. +- Unknown boundary payload: `Schema.decodeUnknownEffect(...)`. +- Service boundary: `Context.Service()(...)` plus `Layer.effect(...)` plus `Service.of(...)`. +- Public or non-trivial internal service method: `Effect.fn("Domain.operation")`. +- Runtime configuration: `Config` recipes read in layers; override with `ConfigProvider` in tests. +- Event source: `Stream` consumed with `Stream.runForEach(...)` and forked with `Effect.forkScoped` in the owning layer. +- Queue-backed event source: `Queue` for the producer boundary, `Stream.fromQueue(...)` for consumers. +- Broadcast event source: `PubSub` / `Stream.fromPubSub(...)` or `SubscriptionRef` for latest-value state. +- Polling worker: `runPass().pipe(Effect.repeat(Schedule.spaced(...)))`, with typed pass failures handled before repeat. +- Retry transient operation: `Effect.retry(...)` / `Effect.retryOrElse(...)` with a bounded `Schedule`. +- Keyed lookup cache with TTL and concurrent-lookup dedupe: prefer `Cache.make(...)` / exit-aware `Cache.makeWith(...)` when their lifecycle and eviction model fit. +- Memoize a single effect result: `Effect.cached(...)` / `Effect.cachedWithTTL(...)`. +- Batch N keys into one backend call (only when a real batch endpoint exists): `Effect.request(...)` + `RequestResolver`. +- HTTP request in an Effect application: prefer Effect `HttpClient` plus request/response schema decoding. +- HTTP transient retry: `HttpClient.retryTransient(...)`. +- Time-sensitive test: `TestClock`, not real sleeping. +- Concurrent/background test synchronization: `Deferred`, `Queue`, `Latch`, `Ref`, or explicit test hooks. + +## Boundary Rules + +- Keep HTTP handlers thin: decode input, read context, call services, map typed errors to transport responses. +- Keep business rules in services or domain functions, not transport handlers. +- Wrap HTTP clients, SDKs, CLIs, and external integrations in named effects at adapter boundaries. +- Decode persisted rows with Schema or SQL-specific helpers when values are not trivially trusted. +- Keep provider/network calls outside authoritative database transactions. +- Catch or retry only when the current boundary has a truthful response. +- Retry only when the operation has proven idempotency. +- Let exhausted failures remain visible unless the boundary has a real fallback. + +## Do Nots + +- Do not use `as any`, non-null assertions, or unchecked casts to silence Effect typing problems. +- Do not introduce `Schema.Class` or `Schema.TaggedClass` as default app data-modeling patterns. +- Do not hand-roll `_tag` error classes when `Schema.TaggedError` fits. +- Do not use cause-level recovery when typed-error recovery is enough. +- Do not use `Layer.mergeAll(...)` or `provideMerge(...)` as blind make-it-compile tools. +- Do not hide required application authority, credentials, persistence, transports, or external services behind `Context.Reference` defaults. +- Do not add arbitrary `Effect.sleep(...)` to tests when a deterministic synchronization primitive is available. +- Do not hand-roll Map/TTL/prune caches or in-flight dedupe when `effect/Cache` fits. diff --git a/.agents/skills/effect/references/CACHING.md b/.agents/skills/effect/references/CACHING.md new file mode 100644 index 0000000000..2454e0aa0d --- /dev/null +++ b/.agents/skills/effect/references/CACHING.md @@ -0,0 +1,75 @@ +# Caching, Memoization, And Request Dedupe + +Use this when memoizing expensive lookups, caching per-key results with TTL, deduplicating concurrent identical calls, or considering request batching. + +Prefer `effect/Cache` over a `Map` + timestamp + prune-loop cache when its keyed memoization, TTL, capacity, lifecycle, and eviction semantics fit. + +## Core Rules + +- `Cache.make({ capacity, lookup, timeToLive })` caches per-key lookups with one fixed TTL for all entries. +- `Cache.makeWith(lookup, { capacity, timeToLive(exit, key) })` computes TTL per entry from the lookup's `Exit` — the tool for "cache successes, not failures". +- Concurrent `Cache.get` calls for the same missing key share one pending lookup — dedupe is built in; do not add your own in-flight tracking. +- `capacity` is required and bounds the cache; stop writing manual prune/evict loops. +- Return a zero TTL (`0` or `"0 millis"`) from `timeToLive` to avoid caching transient failures or degraded fallbacks without failing the caller. A short negative-cache TTL can be appropriate for stable failures such as not-found results. +- `Cache.invalidate(cache, key)` / `Cache.refresh(cache, key)` handle explicit staleness; `Cache.has` checks without triggering a lookup. +- Cache construction is effectful. Build the cache once in the owning layer/scope and share the handle; a cache built per call caches nothing. +- For a single value (no key), use `Effect.cached(effect)` or `Effect.cachedWithTTL(effect, ttl)` instead of a one-key Cache. +- For cached resources that need cleanup (connections, clients), use `ScopedCache`. + +## Exit-Aware TTL (cache successes, skip degraded results) + +```ts +import { Cache, Duration, Effect, Exit } from "effect"; + +const makeResolver = Effect.gen(function* () { + const cache = yield* Cache.makeWith( + (channelRef: string) => resolveUncached(channelRef), // never-failing, returns { where, cacheable } + { + capacity: 300, + timeToLive: (exit) => + Exit.isSuccess(exit) && exit.value.cacheable ? "10 minutes" : Duration.zero, + }, + ); + return (channelRef: string) => + Cache.get(cache, channelRef).pipe(Effect.map((resolved) => resolved.where)); +}); +``` + +This replaces a hand-rolled `Map` plus prune logic, and upgrades it: repeated rows pointing at the same key during one burst share a single provider call. + +## Expensive Client Acquisition Belongs In The Layer, Not The Lookup + +A cache cannot fix a lookup that pays a scoped acquisition per call, such as SDK client construction or authentication. Acquire clients once via the owning layer (`Layer.build` inside a `Layer.unwrap(Effect.gen(...))` composition, or a service dependency) so the cached lookup is a plain call: + +```ts +// Bad: every cache miss acquires a fresh client +const lookup = (id: string) => getRecord(id).pipe(Effect.provide(apiClientLayer(options))); + +// Good: client built once for the layer's lifetime; misses are one API call +// Layer.build requires Scope.Scope; acquire this inside the owning layer's scope. +const context = yield * Layer.build(apiClientLayer(options)); +const lookup = (id: string) => Context.get(context, ApiClient).getRecord(id); +``` + +## Request Batching (`Effect.request` + `RequestResolver`) + +Batching exists for backends with a real batch endpoint: the resolver receives an array of pending requests and can collapse them into one wire call. + +- Use it when the API can answer N keys in one call (SQL `IN (...)`, DataLoader-style endpoints, batch GET). +- Do not reach for it when the backend only has per-item endpoints (most REST provider APIs): a batched resolver still loops one call per entry, so it buys nothing over `Effect.forEach(items, f, { concurrency })` plus `Cache` for dedupe/memoization. +- `RequestResolver.batchN(resolver, n)` bounds batch size; `RequestResolver.makeGrouped` groups requests that must resolve through different targets. + +Selection guide: + +- Same key requested repeatedly over time → `Cache`. +- Same key requested concurrently in one burst → `Cache` (shared pending lookup). +- Many distinct keys, backend has a batch endpoint → `Effect.request` + `RequestResolver`. +- Many distinct keys, per-item endpoint only → `Effect.forEach(..., { concurrency: n })`, optionally through a `Cache`. + +## Do Nots + +- Do not hand-roll Map/TTL/prune caches, in-flight dedupe maps, or LRU logic when `Cache` fits. +- Choose failure TTLs by semantics. Skip transient failures and degraded fallbacks by default; bounded negative caching can protect an upstream from repeated stable failures. +- Do not build a cache inside the request handler or per call — hoist it to the owning layer. +- Do not adopt `RequestResolver` batching for per-item REST endpoints just because "batching" sounds faster. +- Do not put scoped client acquisition inside the cache lookup; acquire once in the layer. diff --git a/.agents/skills/effect/references/CONFIG.md b/.agents/skills/effect/references/CONFIG.md new file mode 100644 index 0000000000..50c03fca7e --- /dev/null +++ b/.agents/skills/effect/references/CONFIG.md @@ -0,0 +1,60 @@ +# Config + +Use this when reading runtime configuration, env vars, `.env` files, provider-specific settings, or writing `layerConfig(...)` helpers. + +Read runtime configuration through Effect `Config` recipes and provider layers, not direct `process.env` access inside application logic. + +```ts +export const dataDirectoryConfig = Config.schema(AbsolutePath, "APP_DATA_DIR"); + +export const layerFromEnvironment = Layer.effect( + Configuration.Service, + Effect.gen(function* () { + const apiKey = yield* Config.redacted("API_KEY"); + const optionalModel = yield* Config.option(Config.string("MODEL")); + const enabled = yield* Config.boolean("FEATURE_ENABLED").pipe(Config.withDefault(false)); + + return Configuration.Service.of({ apiKey, optionalModel, enabled }); + }), +); +``` + +## Config Recipes + +- `Config` is yieldable and reads the current `ConfigProvider` reference. +- The default provider is `ConfigProvider.fromEnv()`. +- Use `Config.redacted(...)` for credentials. +- Use `Config.schema(...)` or `Config.mapOrFail(...)` for refined values. +- Use `Config.option(...)` for semantic absence. +- Use `Config.withDefault(...)` for missing-data defaults only; malformed values still fail. +- Use `Config.orElse(...)` only when intentionally catching any config parse failure. +- Use `Config.unwrap(...)` / `Config.Wrap` for `layerConfig(...)` helpers. + +## Providers + +- Use `ConfigProvider.layer(provider)` to replace the active provider for an app or suite. +- Use `ConfigProvider.layerAdd(provider)` for fallbacks; pass `{ asPrimary: true }` when the added provider must override the current provider. +- Use `ConfigProvider.fromUnknown(...)` for deterministic test config. +- Use `ConfigProvider.fromEnv(...)` for environment variables. +- Use `ConfigProvider.constantCase` when camelCase schema keys should read `SCREAMING_SNAKE_CASE` env vars. +- Use `ConfigProvider.nested(...)` to scope a provider under a prefix. +- Treat `.env`, directory, and environment providers as startup/boundary sources, not business-workflow reads. + +## Layer Config Helpers + +Library-style layers often expose both concrete `layer(options)` and config-backed `layerConfig(options: Config.Wrap)`. + +```ts +export const layerConfig = (config: Config.Wrap) => + Layer.effect( + Client.Service, + Config.unwrap(config).pipe( + Effect.flatMap(makeClient), + Effect.map((client) => Client.Service.of(client)), + ), + ); +``` + +Use this pattern when a service naturally supports runtime config while still allowing tests to pass concrete values. + +Use `Layer.succeed(AppConfiguration.Service, testConfig)` when the app already wraps environment config in an application service and the test does not need to exercise Config decoding itself. diff --git a/.agents/skills/effect/references/HTTP_CLIENTS.md b/.agents/skills/effect/references/HTTP_CLIENTS.md new file mode 100644 index 0000000000..34b6b8033c --- /dev/null +++ b/.agents/skills/effect/references/HTTP_CLIENTS.md @@ -0,0 +1,101 @@ +# HTTP Clients + +Use this when writing outgoing HTTP calls, Effect HttpClient adapters, status classification, HTTP retries, or rate limiting. + +Use Effect HTTP client modules for outgoing HTTP in app/provider code: + +- `effect/unstable/http/HttpClient` +- `effect/unstable/http/HttpClientRequest` +- `effect/unstable/http/HttpClientResponse` +- `effect/unstable/http/HttpClientError` + +Prefer Effect HttpClient in Effect application and provider code when its typed errors, layers, and transforms are useful. Raw `fetch` remains reasonable for browser or edge constraints, small adapters, platform transports, and libraries that intentionally avoid unstable Effect HTTP APIs. + +## Boundary Shape + +HTTP adapter methods should be named effects that own the full boundary: + +- construct request +- attach auth and headers +- execute request +- classify status +- decode response body +- map transport/status/decode failures to typed domain errors +- apply retry/rate-limit policy where idempotent + +Keep raw provider/network effects outside business services and database transactions. + +## Effect HttpClient + +Useful APIs: + +- `HttpClient.get(...)`, `post(...)`, `put(...)`, `patch(...)`, `del(...)`, `execute(...)` for service accessors. +- `HttpClient.mapRequest(...)` / `mapRequestEffect(...)` for configured client transforms. +- `HttpClientRequest.prependUrl(...)` for base URLs. +- `HttpClientRequest.bearerToken(...)` for bearer auth. +- `HttpClientRequest.acceptJson` for JSON accept headers. +- `HttpClientRequest.bodyJson(...)` for effectful JSON body encoding. +- `HttpClientRequest.schemaBodyJson(...)` for schema-backed JSON body encoding. +- `HttpClient.filterStatusOk` / `HttpClientResponse.filterStatusOk` before decoding when non-2xx responses are failures. +- `HttpClientResponse.schemaBodyJson(...)` for body-only decoding, `schemaJson(...)` for status/headers/body decoding, and `schemaNoBody(...)` for status/headers decoding. +- `HttpClient.retryTransient(...)` for common transient HTTP failures. +- `HttpClient.withRateLimiter(...)` for proactive pacing and learning from rate-limit headers. It requires a `RateLimiter` plus initial window, limit, and key options; it adds `RateLimiterError` to the error channel and retries `429` responses by default. + +## Retry And Rate Limits + +Use `HttpClient.retryTransient(...)` for common transient HTTP failures: + +- transport errors +- timeouts +- `408` +- `429` +- `500` +- `502` +- `503` +- `504` + +Use `HttpClient.withRateLimiter(...)` when the client should proactively pace requests and learn from rate-limit / `Retry-After` headers. + +Use operation-level `Effect.retry(...)` when retry depends on domain-specific typed errors, provider payloads, or idempotency rules. Read `SCHEDULING.md` for custom schedules and `retryAfterMs` typed-provider patterns. + +## Raw Fetch Exception + +Use raw `fetch` deliberately when implementing a platform transport, adapting an API that cannot use Effect HttpClient, or targeting a runtime/library boundary where the unstable Effect HTTP modules are not an appropriate dependency. + +If a temporary raw `fetch` boundary is unavoidable, keep it inside an adapter service and still use Effect boundary discipline. + +```ts +const request = Effect.fn("Provider.request")(function* (input: RequestInput) { + const response = yield* Effect.tryPromise({ + try: (signal) => fetch(input.url, { signal, headers: input.headers }), + catch: (cause) => new ProviderError({ operation: "Provider.request", cause }), + }); + + if (!response.ok) { + return yield* Effect.fail( + new ProviderRejected({ + operation: "Provider.request", + status: response.status, + }), + ); + } + + const json = yield* Effect.tryPromise({ + try: () => response.json(), + catch: (cause) => new ProviderError({ operation: "Provider.decodeJson", cause }), + }); + + return yield* Schema.decodeUnknownEffect(ResponseSchema)(json).pipe( + Effect.mapError((cause) => new ProviderError({ operation: "Provider.decodeResponse", cause })), + ); +}); +``` + +Guidance: + +- Prefer replacing this with Effect HttpClient before adding more behavior. +- Wire `AbortSignal` from `Effect.tryPromise` into `fetch` when raw fetch is unavoidable. +- Classify HTTP status before decoding successful payloads. +- Decode unknown response bodies with Schema at the boundary. +- Preserve provider evidence needed for diagnosis, but redact secrets and private payloads. +- Apply retry only for idempotent operations. diff --git a/.agents/skills/effect/references/SCHEDULING.md b/.agents/skills/effect/references/SCHEDULING.md new file mode 100644 index 0000000000..c5a08f48a7 --- /dev/null +++ b/.agents/skills/effect/references/SCHEDULING.md @@ -0,0 +1,133 @@ +# Scheduling And Retry + +Use this when writing retries, repeats, polling workers, backoff, jitter, rate-limit-aware policies, timeouts, or pass loops. + +Use `Schedule` for retry, polling, pacing, and repeated background work instead of hand-rolled `while (true)` loops with sleeps. + +## Core Rules + +- `Effect.retry(...)` retries typed failures; defects and interruptions are not retried. +- `Effect.repeat(...)` repeats successful effects; failures stop repetition unless the pass handles them first. +- The source effect runs once before the schedule is stepped. +- `Schedule.recurs(3)` means three retries/repetitions after the initial run. +- `Schedule.spaced(...)` waits after work completes. +- `Schedule.fixed(...)` aligns executions to a cadence. +- Use `Schedule.exponential(...)` or `Schedule.fibonacci(...)` for backoff. +- Add `Schedule.jittered` to avoid synchronized retry storms. +- Use `Schedule.recurs(...)` for a counter schedule or `Schedule.upTo({ times })` to bound a delay schedule. +- Use `Schedule.tap(({ input }) => ...)` to log retry inputs. +- `Schedule.tap(...)` receives the full schedule metadata, including `input`, `output`, and `duration`. +- Use `Schedule.setInputType()` before input-dependent combinators when the input type would otherwise be `unknown`. +- Use `Effect.retryOrElse(...)` when exhausted retries need a fallback/reporting effect. +- Retry only at the narrowest boundary with proven idempotency. +- Exhausted failures should remain visible unless the boundary has a truthful fallback. + +## Polling Workers + +Prefer typed pass failures over cause recovery. + +```ts +const pass = runPass().pipe( + Effect.tapError((error) => Effect.logError("Worker.pass_failed", error)), + Effect.ignore, +); + +const run = pass.pipe(Effect.repeat(Schedule.spaced("1 second"))); +``` + +This shape says expected operational pass failures are logged and the worker continues. Defects still defect and can reach supervision. + +Use cause-level recovery only at supervision boundaries where the policy is truly "report non-interrupt failure and continue". + +```ts +const logNonInterruptCauseAndContinue = (message: string) => + Effect.catchCauseIf( + (cause) => !Cause.hasInterrupts(cause), + (cause) => Effect.logError(message, cause), + ); +``` + +Do not catch causes just to make failures disappear. If only expected typed failures should be recoverable, use `Effect.catchIf(...)`, `Effect.catchFilter(...)`, `Effect.catchTag(...)`, or `Effect.retry(...)` on those typed errors instead. + +## Per-Item Failure Isolation + +For batch workers, catch expected item-level typed failures around each item so one bad item does not stall the batch. + +```ts +yield * + Effect.forEach( + items, + (item) => + processItem(item).pipe( + Effect.tapError((error) => + Effect.logError("Worker.item_failed", error).pipe( + Effect.annotateLogs({ itemId: item.id }), + ), + ), + Effect.ignore, + ), + { discard: true, concurrency: 5 }, + ); +``` + +Only do this when retrying the item later is truthful or skipping the item is the product policy. + +## Reusable Retry Policy + +```ts +const projectionRetrySchedule: Schedule.Schedule = Schedule.exponential( + "100 millis", +).pipe(Schedule.jittered, Schedule.upTo({ times: 5 })); + +const reconcileWithRetry = (target: Target) => + reconcile(target).pipe( + Effect.retryOrElse( + projectionRetrySchedule.pipe( + Schedule.tap(({ input: error }) => + Effect.logWarning("Agent.Projection.reconcile.retrying").pipe( + Effect.annotateLogs({ operation: error.operation }), + ), + ), + ), + (error) => Effect.logError("Agent.Projection.reconcile.stopped", error), + ), + ); +``` + +Use this when the operation is idempotent and retry state is useful for logs or metrics. + +## Rate-Limit-Aware Typed Retry + +For provider errors that carry `retryAfterMs`, let the schedule use the larger of the backoff delay and the provider delay. + +```ts +type RateLimited = { + readonly retryAfterMs?: number | undefined; +}; + +const providerRetrySchedule: Schedule.Schedule = Schedule.exponential( + "200 millis", +).pipe( + Schedule.jittered, + Schedule.upTo({ times: 5 }), + Schedule.setInputType(), + Schedule.passthrough, + Schedule.modifyDelay(({ input, duration }) => + Effect.succeed( + input.retryAfterMs === undefined + ? duration + : Duration.max(duration, Duration.millis(input.retryAfterMs)), + ), + ), +); +``` + +Use this for operation-level retries over typed provider errors. For Effect HttpClient-level 429 handling and proactive pacing, read `HTTP_CLIENTS.md`. + +## Timeouts And Delays + +- Use `Effect.timeout(...)` when the operation has a real deadline. +- Use `Effect.delay(...)` when one operation should start later. +- Use `Effect.sleep(...)` inside production workflows only when sleeping itself is the domain behavior. +- Avoid manual sleep loops; use `Effect.repeat(...)` with `Schedule` for recurring work. +- In tests, use `TestClock` rather than real time. Read `TESTING.md`. diff --git a/.agents/skills/effect/references/SCHEMA.md b/.agents/skills/effect/references/SCHEMA.md new file mode 100644 index 0000000000..150e8ef103 --- /dev/null +++ b/.agents/skills/effect/references/SCHEMA.md @@ -0,0 +1,130 @@ +# Schema And Data Modeling + +Use this when touching data models, DTOs, row schemas, wire contracts, brands, variants, optional fields, or decoders. + +## Records + +Default to `Schema.Struct(...)` plus a same-name `interface`. + +```ts +export const User = Schema.Struct({ + id: UserId, + name: Schema.NonEmptyString, + email: Schema.optionalKey(Schema.String), +}); + +export interface User extends Schema.Schema.Type {} +``` + +Guidance: + +- Add `.annotate({ identifier: "User" })` only when tooling consumes it: HTTP API, RPC, OpenAPI/JSON Schema, docs, diagnostics, or codegen. +- Use `schema.make(...)` when construction is trusted. +- Use `schema.makeEffect(...)` when construction failure should stay in the Effect error channel. +- Decode unknown input at boundaries with `Schema.decodeUnknownEffect(...)` by default. +- Use `Schema.decodeUnknownSync(...)` only in scripts, tests, or startup paths where throwing is acceptable. +- Use `Schema.decodeUnknownOption(...)` only when mismatch details are intentionally discarded. +- Use `Schema.decodeUnknownResult(...)` for pure code that wants explicit success/failure without Effect. + +## Field And Contract Reuse + +Reuse fields directly when contracts are semantically related. + +```ts +export const CreateUserInput = Schema.Struct({ + name: User.fields.name, + email: User.fields.email, +}); + +export const StoredUser = User.pipe( + Schema.fieldsAssign({ + createdAt: Schema.DateTimeUtcFromString, + }), +); +``` + +Guidance: + +- Use `.fields`, `Schema.fieldsAssign(...)`, and `.mapFields(...)` when contracts are genuinely related. +- Use `Schema.encodeKeys(...)` when decoded TypeScript names differ from encoded wire/storage keys and naming is the only difference. +- Keep explicit mapping when behavior, joins, validation, or domain translation is involved. +- Use `Schema.extendTo(...)` sparingly for decoded-only derived fields. +- Use field reuse to build small related contracts, not one oversized inheritance-by-schema object. + +## Optionality And Defaults + +- Use `Schema.optionalKey(...)` for absent JSON/storage keys. +- Use `Schema.optional(...)` only when explicit `undefined` is part of the contract. +- Use `Schema.NullOr`, `Schema.UndefinedOr`, or `Schema.NullishOr` only when nullish values are truly part of the encoded contract. +- Keep normalized defaulted values as required fields and apply defaults in constructors/decoding. +- Do not make domain values optional merely for construction convenience. + +## Nominal Values + +- Use constrained branded schemas for scalar IDs and value objects. +- Use normal schema constraints before `Schema.brand(...)` for most code. +- Reach for `Schema.fromBrand(...)` only when the project already models brands with `Brand` constructors or wants the check packaged with the brand constructor. + +## Variants + +```ts +type Step = Data.TaggedEnum<{ + Continue: { readonly cursor: number }; + Finished: { readonly count: number }; +}>; + +export const Step = Data.taggedEnum(); + +const next = Step.Continue({ cursor: 10 }); +const label = Step.$match(next, { + Continue: ({ cursor }) => `continue at ${cursor}`, + Finished: ({ count }) => `finished ${count}`, +}); +``` + +```ts +export const Event = Schema.TaggedUnion({ + Started: { runId: RunId }, + Finished: { runId: RunId, result: Schema.Json }, +}); + +export type Event = typeof Event.Type; + +const event = Event.cases.Started.make({ runId }); +const label = Event.match(event, { + Started: ({ runId }) => `started ${runId}`, + Finished: ({ runId }) => `finished ${runId}`, +}); +``` + +Guidance: + +- Use `Data.TaggedEnum` for internal control-flow algebras; it provides constructors, `$is`, and exhaustive `$match`. Do not add a Schema solely to obtain these utilities. +- Use `Schema.TaggedStruct` for the ordinary Effect-owned `_tag` variant. +- Use `Schema.TaggedUnion` when the union needs decoding, encoding, persistence, wire validation, JSON Schema derivation, or schema composition. +- Prefer a principled split over forcing one representation everywhere: Data internally, Schema at boundaries. +- Use `Schema.tag(...)` when an external contract has a custom discriminator field such as `type` or `kind`; combine those structs with `Schema.toTaggedUnion("type")` when union helpers are needed. +- If the encoded contract omits the discriminant, use `Schema.tagDefaultOmit(...)` deliberately. +- Avoid `Schema.Class` and `Schema.TaggedClass` for new data models. + +## Errors + +`Schema.TaggedError` is the explicit class exception for typed Effect errors. + +```ts +export class PersistenceError extends Schema.TaggedError()( + "UserRepo.PersistenceError", + { + operation: Schema.String, + cause: Schema.Defect(), + }, +) {} +``` + +Guidance: + +- Map infrastructure failures into domain-specific tagged errors at service boundaries. +- Include operation labels when they help diagnose adapter, persistence, provider, or transport failures. +- Use schema unions for public API or transport error surfaces. +- Use `Schema.Defect()` for defect-like payloads. +- Preserve interruption when catching broad causes at ingress, worker, or stream boundaries. diff --git a/.agents/skills/effect/references/SERVICES_LAYERS.md b/.agents/skills/effect/references/SERVICES_LAYERS.md new file mode 100644 index 0000000000..50b4c88bf8 --- /dev/null +++ b/.agents/skills/effect/references/SERVICES_LAYERS.md @@ -0,0 +1,155 @@ +# Services, Layers, And Modules + +Use this when defining service tags, module surfaces, layer implementations, runtime wiring, typed errors, or `Effect.fn` operation boundaries. + +## Module Surface + +One opinionated application-module style uses file-local role names and one canonical ES module namespace projection. Follow the existing codebase's module style when it has one; this convention is not required by Effect. + +```ts +export interface Interface { + readonly get: (id: UserId) => Effect.Effect; +} + +export class Service extends Context.Service()("@app/UserRepo") {} + +export const layer = Layer.effect( + Service, + Effect.gen(function* () { + const sql = yield* SqlClient.SqlClient; + + const get = Effect.fn("UserRepo.get")(function* (id: UserId) { + // ... + }); + + return Service.of({ get }); + }), +); + +export class NotFound extends Schema.TaggedError()("UserRepo.NotFound", { id: UserId }) {} + +export * as UserRepo from "./user-repo.js"; +``` + +Consumers use the module namespace. + +```ts +import { UserRepo } from "./user-repo.js"; + +const program = Effect.gen(function* () { + const repo = yield* UserRepo.Service; + return yield* repo.get(id); +}); +``` + +The self-export is deliberate. It lets the file remain the module while giving every consumer the same domain-first name, without a TypeScript `namespace`, wrapper object, or repeated consumer-side aliases. + +```ts +// Sibling module: import the owning leaf directly. +import { UserRepo } from "./user-repo.js"; + +// Folder or package barrel: relay the identity established by the leaf. +export { UserRepo } from "./user-repo.js"; +``` + +Guidance: + +- Do not name the tag class `UserRepo` inside `user-repo.ts`; the module namespace is the domain name. +- In this module style, single-file modules self-export their canonical namespace at the bottom: `export * as UserRepo from "./user-repo.js"`. +- Sibling modules import that namespace from the owning leaf; they do not import through their own aggregate barrel. +- Folder and package barrels relay established leaf identities with `export { UserRepo } from "./user-repo.js"`. +- The resulting `UserRepo.UserRepo === UserRepo` self-reference is unusual. Use this pattern only where the runtime and toolchain support it; otherwise use named exports or a separate barrel. +- Export only intentional surface; keep local schemas, row codecs, helpers, and implementation details unexported. +- Do not introduce TypeScript `namespace` declarations for organization. +- Use a named service class such as `class UserRepo extends Context.Service...` when an external library or existing codebase does not use module namespace style. + +## Layer Constructors + +Choose the layer constructor that matches the thing produced. + +```ts +Layer.succeed(Service, impl); // already-built service +Layer.sync(Service, () => impl); // lazy synchronous service +Layer.effect(Service, makeEffect); // effectful service acquisition +``` + +Guidance: + +- Default real implementations to `Layer.effect(Service, Effect.gen(...))`. +- Use `Layer.effectContext(...)` when one acquisition intentionally supplies multiple services, especially first-class test stubs or one client backing several service tags. +- Use `Layer.unwrap(...)` when config or runtime discovery chooses/builds the layer. +- Use `Layer.fresh(...)` or `Effect.provide(layer, { local: true })` only when a test or operation needs isolated acquisition. +- Use `Context.Reference` rarely, only for ambient/defaultable runtime references where a safe default is real. + +## Long-Lived Work + +A layer that starts a stream, listener, worker, subscription, or forever loop must fork that work into the layer scope. Layer acquisition must complete. + +```ts +export const layer = Layer.effectDiscard( + Effect.gen(function* () { + const events = yield* Events.Service; + + yield* events.stream.pipe(Stream.runForEach(handleEvent), Effect.forkScoped); + }), +); +``` + +Guidance: + +- Use `Effect.forkScoped`, `FiberSet`, or `FiberMap` for scoped background work. +- Do not run forever work inline during layer acquisition. +- Do not expose public `start` methods unless the domain explicitly needs manual lifecycle control. + +## Runtime Wiring + +- Use `Layer.provide(...)` to hide an implementation dependency. +- Use `Layer.provideMerge(...)` only when the dependency should remain exposed for downstream consumers. +- Use `Layer.mergeAll(...)` for independent exposed layers. +- Prefer flat, topologically sorted runtime layer values with named subgraphs. +- Avoid using `provideMerge` as a blind make-it-compile tool. +- Avoid hiding important authority or lifecycle dependencies behind broad invisible provisioning. + +## Effect.fn + +Use extra `Effect.fn(...)` arguments for wrappers that apply to the whole function call. Each transform receives `(effect, ...originalArgs)`. + +```ts +const readAttachment = Effect.fn("Attachment.read")( + function* (ref: AttachmentRef) { + return yield* api.read(ref); + }, + (effect, ref) => effect.pipe(attachmentError("Attachment.read", { attachmentId: ref.id })), +); +``` + +Good whole-function transforms: + +- error classification +- localized recovery +- logging annotations +- spans +- retry +- timeout +- ensuring cleanup +- small local provisioning +- result mapping + +Guidance: + +- Keep the generator body focused on the core workflow. +- Use transforms when the wrapper needs original arguments. +- Do not build long clever pipelines; one or two transforms is usually enough. +- Do not use this for local branch-level handling inside the workflow. + +## Operation Error Helpers + +For boundary errors with operation labels, prefer a shared curried `mapError` helper over hand-writing wrappers in every module. + +```ts +const persistenceError = operationError(PersistenceError.make); + +const row = yield * query.pipe(persistenceError("UserRepository.findById")); +``` + +Name the local helper after the error it produces, such as `persistenceError`, `projectionError`, or `processingError`. Use `Effect.fn(...)` and spans for observability in addition to payload labels, not instead of them. diff --git a/.agents/skills/effect/references/STREAMS.md b/.agents/skills/effect/references/STREAMS.md new file mode 100644 index 0000000000..45b26fdf1f --- /dev/null +++ b/.agents/skills/effect/references/STREAMS.md @@ -0,0 +1,134 @@ +# Streams + +Use this when working with `Stream`, event sources, async iterables, queue/pubsub-backed streams, pagination, backpressure, throttling, debouncing, or long-lived stream consumers. + +## Mental Model + +`Stream` is an effectful source that can emit many `A` values over time, fail with `E`, and require services `R`. Streams are pull-based and backpressured; consumption controls demand. + +Use streams for sources that are naturally many-valued and time-ordered: + +- gateway events +- provider callbacks adapted through queues +- subscription/event logs +- paginated APIs +- file/stdin/platform streams +- scheduled ticks when values matter +- pipelines with filtering, mapping, buffering, throttling, or bounded concurrent processing + +Do not use streams just to loop forever. For one repeated effect with no emitted values, use `Effect.repeat(...)` with `Schedule`; read `SCHEDULING.md`. + +## Source Chooser + +- In-memory values: `Stream.make(...)` or `Stream.fromIterable(...)`. +- Test fixtures: `Stream.fromIterable(...)`, often with `Stream.concat(Stream.never)` for an open subscription. +- Queue-backed callback boundary: `Queue` plus `Stream.fromQueue(...)`. +- Broadcast events: `PubSub` plus `Stream.fromPubSub(...)`. +- Latest-value state plus updates: `SubscriptionRef`. +- Schedule-generated ticks/values: `Stream.fromSchedule(...)`. +- Paginated pull APIs: `Stream.paginate(...)`; its effectful step returns `Effect, Option], E, R>`. The stream emits `A` elements, not page arrays. `Option.none()` ends pagination after emitting that step's elements; empty arrays are allowed. +- Async iterable/platform source: `Stream.fromAsyncIterable(...)` when no native Effect source exists. +- Effect that produces a stream after reading services/config: `Stream.unwrap(...)`. + +## Transformation Chooser + +- Pure transformation: `Stream.map(...)`. +- Effectful transformation: `Stream.mapEffect(...)`. +- Bounded concurrent effectful transformation: `Stream.mapEffect(fn, { concurrency })`. +- Drop ordering when order is irrelevant and latency matters: `Stream.mapEffect(fn, { concurrency, unordered: true })`. +- One input to zero/many outputs: `Stream.flatMap(...)`. +- Multiple inner streams concurrently: `Stream.flatMap(fn, { concurrency })`. +- Keep only matching values: `Stream.filter(...)` / `Stream.filterEffect(...)`. +- Stateful transformation: `Stream.mapAccum(...)` / `Stream.mapAccumEffect(...)`. + +## Consumption Chooser + +- Side-effecting consumer: `Stream.runForEach(...)`. +- Ignore elements but run the stream: `Stream.runDrain`. +- Tests/small finite streams: `Stream.runCollect`. +- First N values in tests: `Stream.take(n)` plus `Stream.runCollect`. +- Fold into a value: `Stream.runFold(...)`. +- Long-lived consumer in a layer: `stream.pipe(Stream.runForEach(...), Effect.forkScoped)`. + +Avoid `Stream.runCollect` on unbounded or production event streams. + +## Long-Lived Consumers + +Own long-lived stream consumers in layers and fork them into the layer scope. + +```ts +export const layer = Layer.effectDiscard( + Effect.gen(function* () { + const gateway = yield* Gateway.Service; + + yield* gateway.events.pipe( + Stream.filter(isMessageEvent), + Stream.runForEach(handleEvent), + Effect.forkScoped, + ); + }), +); +``` + +Guidance: + +- Let the layer own the stream lifetime. +- Use `Effect.forkScoped` for the ordinary case. +- If methods need to fork work into the layer lifetime, capture `Scope.Scope` during layer acquisition and use `Effect.forkIn(scope)` internally. Do not expose the scope as public service API. +- Preserve stream failures unless the owning boundary has a truthful recovery policy. + +## Queues, PubSub, And SubscriptionRef + +- Use `Queue` when each event/item should be consumed by one consumer or worker. +- Use `PubSub` when every subscriber should see every event. +- Use `SubscriptionRef` when consumers need the current value and a stream of changes. +- Expose a `Stream` from service interfaces when callers should consume events, not push into the queue. +- Keep producer queues/private refs inside the implementation or test service. + +Good service shape: + +```ts +export interface Interface { + readonly events: Stream.Stream; + readonly status: Stream.Stream; +} +``` + +Implementation can use private `Queue` / `SubscriptionRef`; consumers see streams. + +## Backpressure And Buffers + +Prefer natural stream backpressure first. + +Use `Stream.buffer(...)` only when producer and consumer should decouple. + +- `strategy: "suspend"`: apply backpressure when full. +- `strategy: "dropping"`: drop new values when full. +- `strategy: "sliding"`: keep the latest values by dropping old ones. +- `capacity: "unbounded"`: rare; use only when growth is bounded elsewhere. + +Use `Stream.debounce(...)` for quiet-period behavior and `Stream.throttle(...)` / `Stream.throttleEffect(...)` for rate-shaped streams. + +## Error Handling + +- Prefer typed stream errors over defects. +- Use `Stream.mapError(...)` to translate errors at boundaries. +- Use `Stream.catchIf(...)`, `Stream.catchTag(...)`, or `Stream.catchFilter(...)` for typed recovery. +- Use `Stream.catchCause(...)` only at explicit supervision boundaries. +- Do not hide stream defects by default; let them reach the owning layer/runtime unless the stream is explicitly best-effort. + +## Keyed Concurrency + +For streams of work keyed by session/channel/id, prefer a named helper over ad hoc maps of fibers. + +If the codebase already has a keyed-run helper (for example a `runForEachKeyed` that runs different keys concurrently while serializing each key and coalescing pending values into one latest-value rerun), use it. Otherwise build one named helper with `FiberMap` rather than scattering fiber bookkeeping through consumers. + +Use this for projection/reconciliation streams where each key needs ordered processing but different keys can run in parallel. + +## Tests + +- Use `Stream.fromIterable(...)` for finite fixtures. +- Use `Stream.empty` for no events. +- Use `Stream.fromQueue(...)` with a test-owned `Queue` when the test needs to drive events interactively. +- Use `Stream.take(n)` plus `Stream.runCollect` for finite assertions. +- Avoid real sleeps; coordinate with `Deferred`, `Queue`, `Latch`, and `TestClock`. diff --git a/.agents/skills/effect/references/TESTING.md b/.agents/skills/effect/references/TESTING.md new file mode 100644 index 0000000000..c65c7a37af --- /dev/null +++ b/.agents/skills/effect/references/TESTING.md @@ -0,0 +1,111 @@ +# Testing + +Use this when writing Effect tests, tests involving time, retry, schedules, concurrency, workers, services, fakes, or config. + +## Defaults + +- Use `it.effect` by default. +- Use `it.live` only when real time or live runtime services are the behavior under test. +- Use test layers and `ConfigProvider` rather than global mutation. +- Use `TestClock.setTime` / `TestClock.adjust` for sleeps, schedules, retries, leases, and timeouts. +- Fork sleeping effects before advancing `TestClock`. +- Avoid arbitrary `Effect.sleep(...)` in tests; it usually makes tests slow and flaky. +- Assert typed failures, rollback, interruption, finalization, retry bounds, idempotency, concurrency laws, and malformed persistence where relevant. + +```ts +it.effect("finds a user", () => + Effect.gen(function* () { + const users = yield* UserRepo.Service; + const result = yield* users.find(UserId.make("u1")); + expect(Option.isSome(result)).toBe(true); + }).pipe(Effect.provide(UserRepo.testLayer)), +); +``` + +## Synchronization Instead Of Sleeps + +- Use `Deferred` for one-shot readiness/completion signals. +- Use `Queue` for handing test-controlled work or observed events across fibers. +- Use `Latch` for reusable open/close coordination gates. +- Use `Ref` for shared test observation state. +- Use explicit test hooks when the production boundary can expose a deterministic synchronization point. + +```ts +it.effect("publishes exactly once", () => + Effect.gen(function* () { + const published = yield* Queue.unbounded(); + const ready = yield* Deferred.make(); + + const runWorker = makeWorker({ + onReady: () => Deferred.succeed(ready, undefined), + onPublish: (message) => Queue.offer(published, message), + }); + + yield* runWorker.pipe(Effect.forkScoped); + + yield* Deferred.await(ready); + const message = yield* Queue.take(published); + + expect(message).toEqual(expectedMessage); + }), +); +``` + +## First-Class App Test Stubs + +Use `TestInterface extends Interface`, `TestService`, and `testLayer` for reusable/stateful fakes. + +```ts +export interface Interface { + readonly send: (message: Message) => Effect.Effect; +} + +export class Service extends Context.Service()("@app/Notifier") {} + +export interface TestInterface extends Interface { + readonly sentMessages: () => Effect.Effect>; + readonly failNextSend: (error: SendError) => Effect.Effect; +} + +export class TestService extends Context.Service()( + "@app/Notifier/Test", +) {} + +export const testLayer = Layer.effectContext( + Effect.gen(function* () { + const sent = yield* Ref.make>([]); + const nextFailure = yield* Ref.make>(Option.none()); + + const service = TestService.of({ + send: Effect.fn("Notifier.Test.send")(function* (message) { + const failure = yield* Ref.getAndSet(nextFailure, Option.none()); + if (Option.isSome(failure)) return yield* Effect.fail(failure.value); + yield* Ref.update(sent, (messages) => [...messages, message]); + }), + sentMessages: Effect.fn("Notifier.Test.sentMessages")(function* () { + return yield* Ref.get(sent); + }), + failNextSend: Effect.fn("Notifier.Test.failNextSend")(function* (error) { + yield* Ref.set(nextFailure, Option.some(error)); + }), + }); + + return Context.empty().pipe(Context.add(Service, service), Context.add(TestService, service)); + }), +); +``` + +Guidance: + +- The same object should back both the real `Service` tag and `TestService` tag. +- Production code depends only on the real service tag. +- Tests use `TestService` for control and inspection. +- Use function-valued service members, including zero-argument operations, so `Effect.fn` fits naturally. +- Use `Layer.succeed` for complete dead-simple static test implementations. +- Use `Layer.mock` only for tiny local partial mocks where omitted members should fail loudly if used. + +## Config In Tests + +Use `ConfigProvider.layer(ConfigProvider.fromUnknown(...))` when the test should exercise Config decoding. + +Use `Layer.succeed(AppConfiguration.Service, config)` when the app wraps decoded config in its own service and the test does not need to exercise env decoding. diff --git a/.gitattributes b/.gitattributes index c48799578d..1f199bbbdb 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,5 +1,39 @@ -# pnpm 12 parses patch files with a Rust patch parser that rejects a carriage -# return in the `---`/`+++` header lines. Git for Windows checks text files out -# with CRLF by default, which broke `pnpm install` on the Windows release -# smoke-test. Keep patch files LF everywhere. +# pnpm 12 briefly had a bug (pnpm/pnpm#14557) where it parsed patch files with a +# Rust patch parser that rejects a carriage return in the `---`/`+++` header lines. +# Git for Windows checks text files out with CRLF by default, which broke +# `pnpm install` on the Windows release smoke-test. +# This guarantees that we keep patch files LF everywhere. patches/*.patch text eol=lf + +# error-tag-stability.unit.test.ts splits this fixture on newlines to diff it +# against a live-scanned tag set. Pin it to LF so a CRLF checkout on Windows +# can't leave a trailing `\r` on every entry and make every tag mismatch. +apps/cli/src/shared/telemetry/__fixtures__/error-tags.txt text eol=lf + +# Generated artifacts. `linguist-generated` keeps these out of GitHub's language +# statistics and collapses them in pull request diffs -- they are build output to +# be regenerated, not code to review, and together they outweigh the hand-written +# source. The attribute is GitHub-only: git, CI, and local diffs are unaffected. +# Collapsed is not hidden. v1-openapi.yaml and the lockfiles do carry changes +# worth reading -- an upstream API change, a dependency substitution -- and both +# stay one click away in review. +# Regeneration is owned by `go generate` in apps/cli-go and by the `generate` +# turbo task, whose `outputs` in turbo.json are the source of truth for this list. +# Only the generated files are listed, not their hand-written inputs +# (pkg/api/*.cfg.yaml, api/overlay.yaml). apps/docs/content/docs/commands/ is +# generated too but gitignored, so it needs no entry. +apps/cli-go/api/v1-openapi.yaml linguist-generated=true +apps/cli-go/pkg/api/*.gen.go linguist-generated=true +packages/api/src/generated/* linguist-generated=true +packages/api/scripts/openapi-source.json linguist-generated=true +apps/cli/src/shared/feedback/database.types.ts linguist-generated=true +apps/docs/public/cli/config.schema.json linguist-generated=true +apps/docs/public/cli/project-config.schema.json linguist-generated=true + +# Lockfiles. Collapsed rather than hidden -- a dependency change is still one +# click away in review, which is the same treatment GitHub gives package-lock.json. +pnpm-lock.yaml linguist-generated=true +apps/cli-go/go.sum linguist-generated=true + +# Not a language marker -- this is the service image manifest, not a container build. +apps/cli/src/shared/services/Dockerfile linguist-detectable=false diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 6b65e010ab..8b94b670bc 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -7,7 +7,7 @@ **/package.json **/go.mod **/go.sum -/apps/cli-go/pkg/config/templates/Dockerfile +/apps/cli/src/shared/services/Dockerfile /pnpm-lock.yaml /pnpm-workspace.yaml @@ -23,6 +23,10 @@ /.github/scripts/ai-review/** @supabase/cli /.github/ai-review/** @supabase/cli +# The cli-artifacts infra workflow deploys with account-admin credentials from develop; +# keep it under maintainer review rather than the ownerless workflow-files rule above. +/.github/workflows/cli-artifacts-infra.yml @supabase/cli + # Generated code. These ownerless rules override the catch-all above so # CI-green sync PRs (e.g. Management API OpenAPI spec) can be auto-merged. /apps/cli-go/pkg/api/*.gen.go diff --git a/.github/MAINTAINERS.md b/.github/MAINTAINERS.md index eee43c5322..7353c710d6 100644 --- a/.github/MAINTAINERS.md +++ b/.github/MAINTAINERS.md @@ -51,26 +51,78 @@ issue directly (from the GitHub UI, or from the Linear-linked issue). ## `run-ci`: full develop CI on stacked or draft PRs Ready (non-draft) PRs targeting `develop` already get the default suite: Test -(check / unit+integration / e2e), preview CLI packages, and PR-title lint. +(check / unit+integration / e2e) and PR-title lint. Stacked PRs (base is another PR branch) and drafts do **not** get that suite unless they carry the **`run-ci`** label. [`run-ci.yml`](./workflows/run-ci.yml) -then calls Test and preview-package publish as reusable workflows, including -while the PR is still a draft. +then calls Test as a reusable workflow, including while the PR is still a draft. - Add `run-ci` to start (or resume) the suite; remove it to cancel in-progress `run-ci` runs via that workflow's concurrency group. -- Other labels do not start or cancel Test / preview. PR-title lint may - retrigger because that check is cheap. +- Other labels do not start or cancel Test. PR-title lint may retrigger because + that check is cheap. - After a stacked PR is retargeted onto `develop`, push or reopen so the native required checks (`Check code quality`, etc.) populate. The opt-in suite uses different check names (`Test / Check code quality`). -- This is independent of `run-live-e2e-ci`, which opts into the separate - supabox live e2e dispatch. +- This is independent of `run-preview-packages` and `run-live-e2e-ci`. The `run-ci` label must exist as a repository label; create it from **Issues → Labels** if it is missing. +## `run-preview-packages`: on-demand pkg.pr.new preview + +CLI preview packages are large, so they are **not** published on every PR. +Add the **`run-preview-packages`** label to publish via +[`publish-preview-cli-packages.yml`](./workflows/publish-preview-cli-packages.yml) +(any base branch, including drafts). While the label stays on, each subsequent +push re-publishes; remove it to cancel in-progress runs. + +The workflow posts (or updates) a PR comment with an `npx` install command for +the preview. This is independent of `run-ci` and `run-live-e2e-ci`. + +The `run-preview-packages` label must exist as a repository label; create it +from **Issues → Labels** if it is missing. + +## Live e2e coverage and stable releases + +[`Live E2E`](./workflows/live-e2e.yml) exercises managed staging after every push +to `develop`, daily at 06:23 UTC, and on manual dispatch. New `develop` pushes +cancel superseded push runs; nightly and manual runs execute independently. +Nightly runs do not depend on a new beta version: they also detect staging +changes between CLI releases. + +Stable publishing requires a passing live suite for the exact release commit. +The release workflow reuses a verified successful staging run on `develop` for +that commit when available; otherwise it runs the suite before publishing. +Normal promotion fast-forwards that commit from `develop` to `main`. The gate +deliberately queries `develop` runs of `live-e2e.yml`; renaming the workflow +requires updating that selector. Actions API lookup errors and live-test +failures block publication. This also applies to +manual stable releases. Beta publication keeps its existing build and smoke-test +gates. + +Live-test failures and recoveries are sent to the channel configured by +`SLACK_RELEASE_WEBHOOK`, with commit and workflow links. Routine successful runs +stay quiet. GitHub Actions logs contain the test failures; notification delivery +does not determine whether the suite passed. + +PR live coverage remains opt-in through `run-live-e2e-ci`. That label dispatches +the PR commit to the separate Supabox harness, which also has its own nightly +schedule against pinned submodules. A Supabox result does not replace the +managed-staging gate for stable publication. + +The gate and notifier identify the reusable suite by the `Live e2e` job name (or +the exact ` / Live e2e` suffix). The gate also checks the `Run live e2e` step +name. Keep these names aligned with their consumers. Push, scheduled, manual, and stable-gate runs +use separate concurrency groups because they own independent temporary project +sets; this is intentional and does not imply a global concurrency quota. +Notification history inspects at most 25 recent runs of the same workflow and +branch. It suppresses repeated outcomes and results superseded by a newer run +or attempt. Recovery requires a known prior failure. History lookup errors +produce warnings; a confirmed current failure can still be reported if its +prior outcome is unknown. Release failures use the existing release notification +to avoid a second failure alert from the live notifier. + ## Deferred: automatic Linear → GitHub label sync We considered auto-applying `open-for-contribution` when a Linear issue moves out of diff --git a/.github/actions/setup/action.yml b/.github/actions/setup/action.yml index d5c4e069b7..1fdd077f3b 100644 --- a/.github/actions/setup/action.yml +++ b/.github/actions/setup/action.yml @@ -75,6 +75,11 @@ runs: shell: bash env: DEPENDENCY_FIREWALL_TOKEN: ${{ inputs.dependency-firewall-token }} + # Skip husky's hook install: the commit-msg hook is a local dev + # convenience, not a gate for bot-authored commits (e.g. + # propose-release-notes.ts), and CI already enforces scopes on the + # PR title separately. + HUSKY: "0" run: | if [ -z "$DEPENDENCY_FIREWALL_TOKEN" ]; then echo "Dependency Firewall token unavailable; using default npm registry." diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 547e80b469..3b6a0cd801 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -11,10 +11,30 @@ updates: actions-major: patterns: - "*" + exclude-patterns: + - "openai/codex-action" ignore: - dependency-name: "supabase/setup-cli" update-types: - "version-update:semver-major" + - dependency-name: "openai/codex-action" + # NOTE: this ecosystem's `versions` strings are parsed as Ruby + # Gem::Requirement (RubyGems comparator syntax), NOT npm-style semver + # ranges — "1.12.x" is not a wildcard here, it parses as a literal + # (and never-matching) version string, which is why the prior attempt + # at this silently failed to block v1.12 (PR #6541, 2026-09-09). Use + # real comparators instead. + versions: + - ">= 1.12, < 1.13" + # Deliberately pinned to v1.11 in ai-review.yml — v1.12 has two + # confirmed, still-open upstream regressions: a wrapper-level hang + # (openai/codex-action#151) and a runner-killing failure + # (openai/codex-action#160). A grouped Dependabot bump already + # silently reintroduced v1.12 once (PR #6484, 2026-09-07). Scoped to + # the 1.12 line (not a blanket ignore) so Dependabot still proposes + # v1.13+ once a fix ships; evaluate any such proposal in its own + # deliberate PR, checking the upstream changelog/issue tracker first + # — never bundle it into the actions-major group. cooldown: default-days: 7 - package-ecosystem: "gomod" @@ -24,6 +44,8 @@ updates: schedule: interval: "cron" cronjob: "0 0 * * *" + commit-message: + prefix: "chore(cli-go): " groups: go-minor: update-types: @@ -43,9 +65,8 @@ updates: interval: "cron" cronjob: "0 0 * * *" commit-message: - prefix: "fix" - prefix-development: "chore" - include: "scope" + prefix: "fix(misc): " + prefix-development: "chore(misc): " groups: npm-major: patterns: @@ -53,12 +74,12 @@ updates: cooldown: default-days: 7 - package-ecosystem: "docker" - directory: "/apps/cli-go/pkg/config/templates" + directory: "/apps/cli/src/shared/services" schedule: interval: "cron" cronjob: "0 0 * * *" commit-message: - prefix: "fix(docker): " + prefix: "fix(misc): " groups: docker-minor: update-types: diff --git a/.github/scripts/ai-review/post-review.test.ts b/.github/scripts/ai-review/post-review.test.ts index cd1db8890a..0f884b196c 100644 --- a/.github/scripts/ai-review/post-review.test.ts +++ b/.github/scripts/ai-review/post-review.test.ts @@ -25,9 +25,7 @@ import { truncateReviewBody, } from "./post-review.ts"; -// A single hunk touching file.ts lines 10-14 on the new side: line 10 is -// context, line 11 replaces a removed line, 12 is a pure addition, 13-14 are -// trailing context. Hand-computed RIGHT-side anchors: {10, 11, 12, 13, 14}. +// One hunk on file.ts; RIGHT-side anchors: {10, 11, 12, 13, 14}. const SINGLE_HUNK_DIFF = `diff --git a/file.ts b/file.ts index 111..222 100644 --- a/file.ts @@ -41,8 +39,7 @@ index 111..222 100644 context line 14 `; -// Two hunks in the same file: {1,2,3} from the first hunk, {20,21,22} from -// the second (the RIGHT counter resets to each hunk's own header). +// Two hunks in one file; each resets the RIGHT counter to its own header. const MULTI_HUNK_DIFF = `diff --git a/multi.ts b/multi.ts index 1..2 100644 --- a/multi.ts @@ -101,8 +98,7 @@ index 0..6 +line three `; -// A trailing "\ No newline at end of file" marker on both sides must not -// perturb the RIGHT counter: anchors are still {1,2}. +// A trailing "\ No newline at end of file" marker must not perturb the RIGHT counter. const NO_NEWLINE_DIFF = `diff --git a/nonewline.ts b/nonewline.ts index 7..8 100644 --- a/nonewline.ts @@ -115,9 +111,7 @@ index 7..8 100644 \\ No newline at end of file `; -// git appends a literal TAB after a `+++` path that needs quoting (here, -// because it contains a space); the tab must be stripped so anchors key on -// "has space.ts", not "has space.ts\t". +// git appends a literal TAB after a quoted `+++` path; it must be stripped. const TAB_PATH_DIFF = `diff --git a/has space.ts b/has space.ts index 9..a 100644 --- a/has space.ts @@ -127,10 +121,7 @@ index 9..a 100644 +added line `; -// A pure rename (100% similarity) carries no `---`/`+++`/`@@` lines at all, -// followed by a normal file's diff — the parser must not leak state (e.g. a -// leftover `currentFile`) from the header-less rename section into the next -// file. +// A pure rename carries no `---`/`+++`/`@@` lines; state must not leak into the next file's diff. const RENAME_ONLY_THEN_NORMAL_DIFF = `diff --git a/old-name.ts b/new-name.ts similarity index 100% rename from old-name.ts @@ -144,10 +135,7 @@ index 1..2 100644 +added `; -// An added line whose literal content is "++ b/not-a-real-header.ts" appears -// in the diff, prefixed by the diff's own "+", as "+++ b/not-a-real-header.ts" -// — a `+++`-lookalike that must not hijack `currentFile` because it occurs -// inside a hunk, not between a `diff --git` boundary and the first `@@`. +// An added line whose content looks like a `+++` header must not hijack `currentFile`. const PLUS_LOOKALIKE_DIFF = `diff --git a/lookalike.ts b/lookalike.ts index 1..2 100644 --- a/lookalike.ts @@ -828,10 +816,6 @@ describe("buildReviewPayload", () => { }); test("truncates the very first payload's body when it already exceeds the cap with zero comments to fold", () => { - // Not anchorable (line 999 is outside the diff hunk), so this produces a - // body-only payload with no inline comments — the 422-retry fold path - // never runs, so only truncating `buildReviewPayload`'s own body catches - // an oversized initial POST. const finding = makeFinding({ file: "file.ts", line: 999, claim: "x".repeat(70_000) }); const review = makeMergedReview({ findings: [finding] }); const payload = buildReviewPayload(review, anchors, footer); @@ -1045,10 +1029,7 @@ describe("post flow via injected ReviewIo", () => { if (opts.failSupersede) { return Promise.reject(new Error("listReviews failed")); } - // Mirror real GitHub: a review posted earlier in the same run shows - // up in later listings as a marker-bearing bot review. The supersede - // pass must snapshot BEFORE posting or it would wrap the fresh - // review as "superseded" too. + // Mirrors real GitHub: a review posted earlier in the same run shows up in later listings. const alreadyPosted = postedReviews.map((payload, i) => ({ id: 900 + i, body: payload.body, @@ -1141,9 +1122,6 @@ describe("post flow via injected ReviewIo", () => { await postConsolidatedReview(io, 42, review, footer); - // With no prior AI review on the PR, nothing may be wrapped as superseded - // — especially not the review this run just posted (which the fake's - // listReviews, like real GitHub, includes in post-POST listings). expect(postedReviews).toHaveLength(1); expect(updatedReviews).toEqual([]); expect(updatedComments).toEqual([]); @@ -1230,8 +1208,6 @@ describe("post flow via injected ReviewIo", () => { }); test("posts a truncated body on the very first attempt for an oversized body-only review (no comments to fold)", async () => { - // Not anchorable, so there's no inline comment for GitHub to 422 on — the - // old behavior threw here instead of posting a truncated body. const finding = makeFinding({ file: "file.ts", line: 999, claim: "x".repeat(70_000) }); const review = makeMergedReview({ findings: [finding] }); const { io, postedReviews } = makeReviewIo({ diff: SINGLE_HUNK_DIFF }); diff --git a/.github/scripts/ai-review/post-review.ts b/.github/scripts/ai-review/post-review.ts index c56949d1d3..93a042ed44 100644 --- a/.github/scripts/ai-review/post-review.ts +++ b/.github/scripts/ai-review/post-review.ts @@ -1,56 +1,38 @@ /** - * AI review poster: validates the structured findings both model passes - * produce, and posts the ONE consolidated PR review the pipeline is allowed - * to post per run. + * AI review poster: validates the structured findings the two model passes + * produce, and posts the one consolidated PR review this pipeline posts per run. * - * Four subcommands, dispatched from `argv`: - * - `validate-findings ` — checks a Claude findings JSON file against - * the shape `.github/ai-review/findings.schema.json` describes. The - * `--json-schema` flag passed to `claude` is a hint to the model, not a - * runtime guarantee, so the CI step re-checks the extracted output here - * before it is trusted. - * - `validate-merged ` — same idea for the Codex-adjudicated merged - * review, against `.github/ai-review/merged-review.schema.json`. - * - `redact ` — reads a JSON file, deep-walks every string value - * through `redactSecrets`, and writes it back in place. Run on every - * model-output JSON file before it's uploaded as a (public-repo) - * artifact, so a prompt-injected `Read` of a secret-bearing path can't - * smuggle a credential out through the artifact even though the posted - * review is already scrubbed at render time. - * - `post` — snapshots the PR's prior AI reviews, posts the consolidated - * review, THEN best-effort supersedes the snapshotted ones (the - * marker/dedup guard in `resolve.ts` should normally prevent a second - * run, but `/ai-review` lets a maintainer force one). The snapshot must - * happen BEFORE the POST — the fresh review is itself a marker-bearing - * bot review, so a post-hoc listing would sweep it into its own - * supersede pass and every new review would collapse itself. Posting - * before superseding, and treating both the snapshot and the supersede - * as best-effort, means a cosmetic failure can never cost the real - * review. + * Subcommands, dispatched from `argv`: + * - `validate-findings ` / `validate-merged ` — check a model's + * findings/merged-review JSON against its schema under `.github/ai-review/`. + * The `--json-schema` flag passed to `claude` is only a hint to the model, + * so this re-validates its output before it's trusted. + * - `redact ` — scrubs secret-shaped strings from a JSON file in + * place; run on every model-output file before it's uploaded as a + * (public-repo) artifact. + * - `post` — builds and posts the consolidated review, then supersedes any + * prior AI reviews on the PR (see `listPriorRunsBestEffort` for why the + * snapshot must happen before the POST). * * `parseDiffAnchors`, `partitionFindings`, `renderReviewBody`, * `renderInlineComment`, `buildReviewPayload`, `foldInlineCommentsIntoBody`, * `supersededBody`, `isSuperseded`, `sanitizeFilePath`, and `redactSecrets` - * are pure and exported for tests. `postConsolidatedReview` is the I/O - * orchestration function for the `post` subcommand; it's exported so a test can drive it against - * an injected `ReviewIo` fake without the network, the same way - * `resolveDecision` is tested in `resolve.ts`. `main()` wires up the real - * GitHub I/O and argv dispatch. + * are pure and exported for tests. `postConsolidatedReview` drives the `post` + * subcommand against an injected `ReviewIo` fake in tests. * * Run in CI as: `bun .github/scripts/ai-review/post-review.ts `. */ export const AI_REVIEW_MARKER = ""; const SUPERSEDED_SUMMARY = "Superseded by a newer AI review"; -/** Hidden marker `isSuperseded` looks for. Kept out of the human-readable - * `SUPERSEDED_SUMMARY` text and broken by `sanitizeModelText` so a model - * can't forge or evade a supersede by echoing the visible text into a - * `claim`/`summary` field. */ +/** Hidden marker `isSuperseded` checks for. Kept out of the human-readable + * `SUPERSEDED_SUMMARY` text, and broken by `sanitizeModelText`, so a model + * can't forge or evade a supersede by echoing it into a `claim`/`summary` field. */ const SUPERSEDED_MARKER = ""; const WORKFLOW_BOT_LOGIN = "github-actions[bot]"; const GITHUB_REVIEW_BODY_MAX = 65536; -// --- Shared types (mirror the two schema files by hand; keep in sync) --- +// Mirrors the two schema files under `.github/ai-review/`; keep them in sync by hand. export type Severity = "critical" | "major" | "minor" | "nit"; export type Verdict = "confirmed" | "refuted" | "uncertain"; @@ -107,12 +89,8 @@ export interface MergedReview { stats: MergedReviewStats; } -// --- Hand-rolled schema validators --- -// -// `.github/ai-review/findings.schema.json` and `merged-review.schema.json` -// are the model-facing contract (passed as `--json-schema`/`output-schema-file`); -// these validators are the runtime enforcement and must be kept in sync with -// them by hand whenever either shape changes. +// Runtime enforcement for the schemas in `.github/ai-review/`; keep these in +// sync by hand whenever either shape changes. function isRecord(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); @@ -219,12 +197,11 @@ function expectCategory(value: unknown, path: string, context: string): string { return str; } -/** `file` is model-controlled and rendered inside `` `code` `` spans at - * several sites; a backtick, newline, other ASCII control char, or `<` in it - * could break out of the span (markdown/HTML injection, mention/#ref pings) - * or forge one of the hidden HTML-comment markers. Reject those at parse - * time as the primary defense; `sanitizeFilePath` neutralizes the same - * characters again at render time in case a caller ever skips validation. */ +/** `file` is model-controlled and rendered inside `` `code` `` spans; a + * backtick, newline, other ASCII control char, or `<` in it could break out + * of the span or forge a hidden marker, so reject those at parse time. + * `sanitizeFilePath` neutralizes the same characters again at render time in + * case a caller skips validation. */ // eslint-disable-next-line no-control-regex -- matching control characters is the point of this pattern const FILE_PATH_FORBIDDEN_PATTERN = /[`<\x00-\x1f\x7f]/; @@ -385,8 +362,6 @@ export function assertMergedReview(value: unknown): asserts value is MergedRevie parseMergedReview(value); } -// --- Diff anchoring --- - const DIFF_GIT_HEADER = /^diff --git /; const HUNK_HEADER = /^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@/; const NEW_FILE_HEADER = /^\+\+\+ (?:b\/(.+)|\/dev\/null)$/; @@ -408,15 +383,13 @@ function stripTrailingTab(path: string): string { } /** - * Parses a unified diff into, for each file, the set of new-side (RIGHT) line - * numbers present in the diff — i.e. the lines a PR review comment can - * anchor to. Context and `+` lines advance the RIGHT counter and are - * anchorable; `-` lines don't exist on the new side and are skipped. + * Parses a unified diff into, for each file, the new-side (RIGHT) line + * numbers a PR review comment can anchor to. Context and `+` lines advance + * the RIGHT counter; `-` lines don't exist on the new side and are skipped. * - * Tracks whether we're inside a hunk so a `+++ ` file header is only ever - * recognized between a `diff --git` boundary and that file's first `@@` - * hunk — otherwise an added/context line whose literal content happens to - * start with `+++ ` (a `+++`-lookalike) could hijack `currentFile`. + * A `+++ ` file header is only recognized between a `diff --git` boundary and + * that file's first `@@` hunk, so an added/context line that happens to start + * with `+++ ` can't hijack `currentFile`. */ export function parseDiffAnchors(diff: string): Map> { const anchors = new Map>(); @@ -461,8 +434,6 @@ function isAnchorable(anchors: Map>, file: string, line: num return anchors.get(file)?.has(line) ?? false; } -// --- Findings partitioning and rendering --- - export interface PartitionedFindings { /** Confirmed/uncertain findings whose start line lands on a diff hunk; posted as inline comments. */ anchorable: MergedFinding[]; @@ -510,17 +481,14 @@ const HTML_COMMENT_OPENER_PATTERN = /"); } -/** Neutralizes the same characters `expectFile` rejects at parse time - * (backtick, `<`, ASCII control chars) inside a model-provided `file` path - * before it's rendered into a `` `code` `` span. Every finding reaching a - * render site will already have passed `expectFile`; this is defense-in-depth - * for any caller that renders a `MergedFinding` without going through - * `assertMergedReview` first. */ +/** Strips the same characters `expectFile` rejects at parse time (backtick, + * `<`, ASCII control chars) so a caller that renders a `MergedFinding` + * without going through `assertMergedReview` first still can't break a + * rendered `` `code` `` span. */ // eslint-disable-next-line no-control-regex -- matching control characters is the point of this pattern const FILE_PATH_UNSAFE_CHARS = /[`<\x00-\x1f\x7f]/g; @@ -629,9 +588,8 @@ export function renderInlineComment(finding: MergedFinding): string { export interface ReviewFooterInfo { trigger: Trigger; runUrl: string; - /** e.g. `` `claude-fable-5` + `gpt-5.6-sol` ``. Passed in from the workflow's - * `CLAUDE_MODEL`/`CODEX_MODEL` env vars instead of being hardcoded here, so - * the model names have one source of truth. */ + /** e.g. `` `claude-fable-5` + `gpt-5.6-sol` ``. Sourced from the workflow's + * model env vars so there's one source of truth for the model names. */ modelsFooter: string; } @@ -768,10 +726,8 @@ export function buildReviewPayload( const partitioned = partitionFindings(review.findings, anchors); const comments = partitioned.anchorable.map((finding) => buildInlineComment(finding, anchors)); const body = renderReviewBody(review, partitioned, footer); - // A body-only review (many non-anchorable findings, few or no inline - // comments) has no fold-retry path to truncate it on a 422 — truncate the - // very first payload too, so an oversized body posts truncated instead of - // throwing when GitHub rejects it for exceeding the review body cap. + // A body-only review has no 422 fold-retry to truncate it later, so + // truncate the first payload too rather than posting an oversized body. return { event: "COMMENT", body: truncateReviewBody(body, footer.runUrl), comments }; } @@ -791,10 +747,7 @@ export function foldInlineCommentsIntoBody(payload: ReviewPayload): ReviewPayloa } /** Truncates a review body to GitHub's 65536-char review body cap, appending - * an explicit truncation marker + the workflow run URL. Applied to both the - * very first payload (`buildReviewPayload`) and the folded 422-retry body - * (every inline comment stuffed into one body), a no-op when the body is - * already under the cap. */ + * a truncation marker with the workflow run URL. No-op under the cap. */ export function truncateReviewBody(body: string, runUrl: string): string { if (body.length <= GITHUB_REVIEW_BODY_MAX) { return body; @@ -822,8 +775,6 @@ export function supersededBody(oldBody: string): string { ].join("\n"); } -// --- Injected GitHub I/O --- - export interface MarkedEntry { id: number; body: string; @@ -863,12 +814,10 @@ function isSupersedableAiEntry(entry: MarkedEntry): boolean { ); } -/** Snapshots the prior AI reviews/comments to supersede. MUST run before the - * new review is posted: the fresh review is itself a marker-bearing bot - * review, so a post-hoc listing would sweep it into its own supersede pass - * and every new review would immediately collapse as "superseded". - * Best-effort — a listing failure degrades to an empty snapshot (prior runs - * stay unwrapped) rather than costing the real review. */ +/** Snapshots the prior AI reviews/comments to supersede. Must run before the + * new review is posted, or a post-hoc listing would sweep the fresh review + * into its own supersede pass. Best-effort — a listing failure degrades to + * an empty snapshot rather than costing the real review. */ async function listPriorRunsBestEffort(io: ReviewIo, prNumber: number): Promise { try { const [reviews, comments] = await Promise.all([ @@ -886,9 +835,8 @@ async function listPriorRunsBestEffort(io: ReviewIo, prNumber: number): Promise< } /** Wraps the snapshotted prior AI reviews/comments in a superseded `
` - * block. Best-effort: a cosmetic failure here (e.g. a transient 404 on a - * review that was deleted mid-run) must never fail the pipeline after the - * real review has already been posted. */ + * block. Best-effort: a cosmetic failure here must never fail the pipeline + * after the real review has already been posted. */ async function supersedePriorRunsBestEffort( io: ReviewIo, prNumber: number, @@ -945,8 +893,6 @@ export async function postConsolidatedReview( await supersedePriorRunsBestEffort(io, prNumber, prior); } -// --- Real GitHub I/O (only runs when executed directly) --- - function requireEnv(name: string): string { const value = process.env[name]; if (!value) { @@ -995,11 +941,7 @@ function isRecordEntry(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); } -/** - * The validated boundary between `Response.json()` (typed `Promise` - * under `@tsconfig/bun`) and this file's typed shapes: `assert` narrows the - * parsed value to `T` before any caller reads a field off it. - */ +/** Narrows `Response.json()`'s `unknown` result to `T` via `assert` before any caller reads a field off it. */ async function githubJson( response: Response, assert: (value: unknown) => asserts value is T, @@ -1132,9 +1074,7 @@ async function postReview( "application/vnd.github+json", [422], ); - // `githubFetch` only returns without throwing for a 2xx or the allowed - // 422; read the body for the 422 case too so a second failed retry can - // surface it instead of discarding it. + // Read the body for the 422 case too so a second failed retry can surface it. if (response.status === 422) { return { status: response.status, body: await response.text() }; } @@ -1172,9 +1112,6 @@ async function runPost(): Promise { const trigger = parseTrigger(requireEnv("TRIGGER")); const runUrl = requireEnv("RUN_URL"); const mergedReviewPath = requireEnv("MERGED_REVIEW_PATH"); - // Sourced from the workflow's top-level `env:` block (the same values fed - // to the `claude`/`codex-action` invocations), not hardcoded here, so the - // model names have one source of truth. const claudeModel = requireEnv("CLAUDE_MODEL"); const codexModel = requireEnv("CODEX_MODEL"); @@ -1189,8 +1126,7 @@ async function runPost(): Promise { console.log(`Posted AI review on PR #${prNumber} (${raw.findings.length} finding(s)).`); } -/** Reads a JSON file, redacts every string value in place through - * `redactSecretsDeep`, and writes it back — the `redact` subcommand's I/O. */ +/** Reads a JSON file, redacts every string value in place, and writes it back. */ async function runRedact(path: string): Promise { const raw: unknown = JSON.parse(await Bun.file(path).text()); const redacted = redactSecretsDeep(raw); diff --git a/.github/scripts/ai-review/resolve.test.ts b/.github/scripts/ai-review/resolve.test.ts index 0beb52747d..07453c1c32 100644 --- a/.github/scripts/ai-review/resolve.test.ts +++ b/.github/scripts/ai-review/resolve.test.ts @@ -361,8 +361,6 @@ describe("resolveDecision: issue_comment authorization", () => { io, ); expect(result.shouldRun).toBe(true); - // The effective permission is always resolved (only the write-permission - // requirement short-circuits for OWNER), so the lookup still happens. expect(permissionLookups).toEqual(["maintainer"]); expect(reactions).toEqual([555]); }); diff --git a/.github/scripts/ai-review/resolve.ts b/.github/scripts/ai-review/resolve.ts index 400d0e3973..58df368df1 100644 --- a/.github/scripts/ai-review/resolve.ts +++ b/.github/scripts/ai-review/resolve.ts @@ -1,25 +1,19 @@ /** * AI review resolver: decides whether the one-shot AI review pipeline should - * run for a PR. + * run for a PR. The pipeline runs once per PR, so this is the only gate + * between a new commit landing and Claude/Codex burning API budget again. * - * The pipeline runs EXACTLY ONCE per PR, so this is the only gate standing - * between "new commit lands" and "Claude + Codex burn API budget again". Two - * triggers feed it: - * - manual (`workflow_dispatch` or an internal maintainer's `/ai-review` - * issue comment): a human explicitly asked for a review, so the - * marker/dedup guard and the draft/fork/bot skips are bypassed. - * - auto (`pull_request` `opened`/`ready_for_review`): only PRs whose - * author has repository write access get the automatic review. Skips - * drafts, bots, fork PRs, authors without write access (external - * contributors go through the manual maintainer path), and PRs that - * already carry a marker comment/review from a prior run. + * Two triggers: + * - manual (`workflow_dispatch` or a maintainer's `/ai-review` issue + * comment): bypasses the marker/dedup guard and the draft/fork/bot skips. + * - auto (`pull_request` `opened`/`ready_for_review`): only for PRs whose + * author has repository write access; skips drafts, bots, fork PRs, and + * PRs that already carry a marker comment/review from a prior run. * - * `resolveDecision` is the pure orchestration function (I/O injected, like - * `evaluateAllOpenPrs` in `contribution-gate.ts`) that a test can drive - * without the network; `main()` wires up the real GitHub I/O, writes the - * step outputs `should_run`, `pr_number`, `head_ref`, and `trigger` to - * `$GITHUB_OUTPUT`, and surfaces the skip reason (if any) in - * `$GITHUB_STEP_SUMMARY`. + * `resolveDecision` is the pure orchestration function (I/O injected) a test + * can drive without the network; `main()` wires up the real GitHub I/O, + * writes `should_run`/`pr_number`/`head_ref`/`trigger` to `$GITHUB_OUTPUT`, + * and surfaces the skip reason (if any) in `$GITHUB_STEP_SUMMARY`. * * Run in CI as: `bun .github/scripts/ai-review/resolve.ts`. */ @@ -29,14 +23,11 @@ import { appendFileSync } from "node:fs"; import { fetchAuthorPermission, WRITE_PERMISSIONS } from "../contribution-gate.ts"; import { AI_REVIEW_MARKER } from "./post-review.ts"; -// Re-export so existing consumers (tests, this file's own dedup check) can -// keep importing the marker from `resolve.ts`; `post-review.ts` — which owns -// posting — is the single source of truth for the literal. +// `post-review.ts` owns the literal; re-exported here for convenience. export { AI_REVIEW_MARKER }; /** Login every review/comment posted by this workflow carries. Duplicated - * (not imported) from `post-review.ts`'s `WORKFLOW_BOT_LOGIN`; keep the two - * literals in sync. */ + * from `post-review.ts`'s `WORKFLOW_BOT_LOGIN`; keep the two in sync. */ const WORKFLOW_BOT_LOGIN = "github-actions[bot]"; export type EventName = "workflow_dispatch" | "issue_comment" | "pull_request"; @@ -96,11 +87,9 @@ export interface ResolveResult { trigger: Trigger; } -/** No size gate: Claude and Codex review agentically — reading the diff and the - * changed files via their own tools over many turns, like the local CLI — so a - * PR that clears the draft/bot/fork/dedup checks is reviewed regardless of its - * size. Very large diffs are handled best-effort within the model's - * context/turn budget. */ +/** No size gate: Claude and Codex review agentically over many turns, so any + * PR that clears the draft/bot/fork/dedup checks is reviewed regardless of + * size, best-effort within the model's context/turn budget. */ function decideForPr(trigger: Trigger): ResolveResult { return { shouldRun: true, trigger }; } @@ -128,9 +117,8 @@ export async function resolveDecision(input: ResolveInput, io: ResolveIo): Promi throw new Error("issue_comment trigger requires comment details"); } - // Authoritative command match: the workflow's job `if:` only - // pre-filters on `startsWith('/ai-review')`, so `/ai-reviewers` or - // `/ai-review-please` would otherwise also reach here. + // The workflow's job `if:` only pre-filters on `startsWith('/ai-review')`, + // so `/ai-reviewers` or `/ai-review-please` would otherwise also reach here. const firstLine = comment.body.split("\n")[0]?.trim() ?? ""; if (firstLine !== "/ai-review") { return { @@ -140,13 +128,9 @@ export async function resolveDecision(input: ResolveInput, io: ResolveIo): Promi }; } - // Authoritative authorization: always resolve the commenter's - // effective repository permission and require write/admin. Only the - // repository OWNER may short-circuit that requirement — any other - // association (including MEMBER/COLLABORATOR, which merely mean "in - // the org"/"added as a collaborator", not necessarily push-capable) - // must pass the permission check. Mirrors `contribution-gate.ts`'s - // `WRITE_PERMISSIONS`. + // Only the repository OWNER short-circuits the permission check; any + // other association (MEMBER/COLLABORATOR only mean "in the org"/"added + // as a collaborator", not necessarily push-capable) must pass it. const permission = await io.fetchPermission(comment.authorLogin); const authorized = comment.authorAssociation === "OWNER" || @@ -175,8 +159,6 @@ export async function resolveDecision(input: ResolveInput, io: ResolveIo): Promi return decideForPr(trigger); } - // Auto trigger (`pull_request` events): internal PRs only, fires at most - // once per PR. if (pr.draft) { return { shouldRun: false, skipReason: "PR is a draft.", trigger }; } @@ -191,13 +173,10 @@ export async function resolveDecision(input: ResolveInput, io: ResolveIo): Promi }; } - // Authoritative auto-trigger authorization: only PRs authored by someone - // with effective repository write access are reviewed automatically. This - // is the actual author check, not defense-in-depth — a same-repo head - // branch only proves the branch exists in this repo, not that the AUTHOR - // pushed it (a PR can be opened from a branch someone else pushed). An - // unresolvable permission counts as unauthorized. Mirrors the manual - // path's gate above and `contribution-gate.ts`'s `WRITE_PERMISSIONS`. + // A same-repo head branch only proves the branch exists in this repo, not + // that the author pushed it, so this checks the author's own permission + // rather than trusting the branch's origin. An unresolvable permission + // counts as unauthorized. const authorPermission = await io.fetchPermission(pr.authorLogin); if (authorPermission === undefined || !WRITE_PERMISSIONS.has(authorPermission)) { return { @@ -214,9 +193,8 @@ export async function resolveDecision(input: ResolveInput, io: ResolveIo): Promi io.listReviews(pr.number), io.listIssueComments(pr.number), ]); - // Only a marker posted BY the workflow bot counts — otherwise anyone could - // paste the (invisible) marker into a comment to permanently suppress the - // auto review of their own PR. + // Only a marker posted by the workflow bot counts, or anyone could paste + // the invisible marker into a comment to suppress their own PR's review. const alreadyReviewed = [...reviews, ...comments].some( (entry) => entry.authorLogin === WORKFLOW_BOT_LOGIN && entry.body.includes(AI_REVIEW_MARKER), ); @@ -231,8 +209,6 @@ export async function resolveDecision(input: ResolveInput, io: ResolveIo): Promi return decideForPr(trigger); } -// --- GitHub I/O (only runs when executed directly) --- - function requireEnv(name: string): string { const value = process.env[name]; if (!value) { @@ -275,11 +251,7 @@ function isRecordEntry(value: unknown): value is Record { return typeof value === "object" && value !== null && !Array.isArray(value); } -/** - * The validated boundary between `Response.json()` (typed `Promise` - * under `@tsconfig/bun`) and this file's typed shapes: `assert` narrows the - * parsed value to `T` before any caller reads a field off it. - */ +/** Narrows `Response.json()`'s `unknown` result to `T` via `assert` before any caller reads a field off it. */ async function githubJson( response: Response, assert: (value: unknown) => asserts value is T, @@ -389,10 +361,8 @@ async function reactToComment(token: string, base: string, commentId: number): P }); } -/** Writes each `$GITHUB_OUTPUT` value using the heredoc/delimiter form (with - * a random delimiter per line) rather than `name=value`, defensively — none - * of today's values can contain a newline, but a future value shouldn't be - * able to inject extra output lines either. */ +/** Writes each `$GITHUB_OUTPUT` value with a random per-line delimiter rather + * than `name=value`, so a future multi-line value can't inject extra output lines. */ function writeOutputs(result: ResolveResult, prNumber: number): void { const outputFile = requireEnv("GITHUB_OUTPUT"); const entries: Record = { @@ -410,9 +380,7 @@ function writeOutputs(result: ResolveResult, prNumber: number): void { appendFileSync(outputFile, `${lines.join("\n")}\n`); } -/** Surfaces the skip reason (if any) in the job's step summary — the only - * place it's actually read; it's not exposed as a job `outputs:` because - * nothing downstream consumes it there. */ +/** Surfaces the skip reason (if any) in the job's step summary. */ function writeStepSummary(result: ResolveResult): void { if (!result.skipReason) { return; diff --git a/.github/scripts/contribution-gate.test.ts b/.github/scripts/contribution-gate.test.ts index 7951de817a..6c8ed6a544 100644 --- a/.github/scripts/contribution-gate.test.ts +++ b/.github/scripts/contribution-gate.test.ts @@ -21,9 +21,6 @@ describe("isInternalAuthor", () => { ); test.each(["admin", "write"])("treats push-capable permission %s as internal", (permission) => { - // A private org member surfaces only as CONTRIBUTOR/NONE via - // author_association, but their effective repo permission gives them - // away as a maintainer. expect(isInternalAuthor("CONTRIBUTOR", permission)).toBe(true); expect(isInternalAuthor("NONE", permission)).toBe(true); }); @@ -130,8 +127,6 @@ describe("evaluateGate", () => { }); test("ignores an open, labeled issue from a different repository", () => { - // Cross-repo closing keyword (e.g. `Closes attacker/repo#1`): the issue is - // controlled by the contributor, so it must not satisfy the gate. const result = evaluateGate({ repository: REPO, isInternal: false, @@ -231,7 +226,6 @@ describe("evaluateAllOpenPrs", () => { expect(byNumber[6]?.pass).toBe(true); expect(byNumber[6]?.reason).toBe("internal"); expect(closed.find((c) => c.number === 1)?.message).toContain(GATE_LABEL); - // Bots and public members are settled without a permission lookup. expect(permissionLookups).not.toContain("maint"); expect(permissionLookups).not.toContain("dependabot"); }); @@ -275,9 +269,6 @@ describe("fetchAuthorPermission", () => { }); test("maps a 404 (non-collaborator fork author) to undefined", async () => { - // The endpoint 404s for users who are not collaborators — the common case - // for the external contributors the gate targets. It must map to external, - // not abort the run. stubFetch(404, { message: "Not Found" }); const permission = await fetchAuthorPermission("t", "supabase", "cli", "ext"); expect(permission).toBeUndefined(); diff --git a/.github/scripts/contribution-gate.ts b/.github/scripts/contribution-gate.ts index 061fc7107b..5d2666446d 100644 --- a/.github/scripts/contribution-gate.ts +++ b/.github/scripts/contribution-gate.ts @@ -1,13 +1,11 @@ /** - * Contribution gate: enforces the Supabase CLI contribution workflow across all - * OPEN pull requests opened by external contributors. + * Contribution gate: enforces the Supabase CLI contribution workflow across + * every open pull request opened by external contributors. * - * A PR passes only when it links to an OPEN GitHub issue that carries the - * `open-for-contribution` label. Maintainers (recognised by their effective - * repository permission, so private org members count too — not just the - * `author_association` GitHub exposes for public members) and bots are exempt - * (they work from Linear tickets or automation). PRs that do not follow the - * process are commented on and closed. + * A PR passes only when it links to an open GitHub issue that carries the + * `open-for-contribution` label. Maintainers (recognized by their effective + * repository permission, so private org members count too) and bots are + * exempt. PRs that don't follow the process are commented on and closed. * * Two modes, both driven from `main()`: * - single-PR (default): reacts to one PR on `pull_request_target` @@ -16,13 +14,10 @@ * open PR, for on-demand `workflow_dispatch` runs. Set `DRY_RUN=true` to * log decisions without commenting/closing. * - * In both modes the workflow checks out the base branch, so this only ever - * executes trusted repository code — it never runs a fork's code. + * The workflow checks out the base branch in both modes, so this only ever + * executes trusted repository code. * * Run in CI as: `bun .github/scripts/contribution-gate.ts`. - * The pure `evaluateGate` decision and the `evaluateAllOpenPrs` orchestrator - * (I/O injected) are unit-tested in `contribution-gate.test.ts`; `main()` wires - * up the real GitHub I/O. */ export const GATE_LABEL = "open-for-contribution"; @@ -30,25 +25,19 @@ export const GATE_LABEL = "open-for-contribution"; /** * Author associations treated as internal (exempt from the gate). * - * NOTE: `author_association` only reports `MEMBER` when a user's organization - * membership is *public*. A private org member (or a team member who keeps - * their membership private) is reported as `CONTRIBUTOR`/`NONE`, so this set - * alone is not enough to identify internal maintainers — see - * `isInternalAuthor`, which also consults the author's effective repository - * permission. + * `author_association` only reports `MEMBER` for a *public* org membership; a + * private member is reported as `CONTRIBUTOR`/`NONE`, so this set alone can't + * identify every internal maintainer — see `isInternalAuthor`. */ export const INTERNAL_ASSOCIATIONS = new Set(["OWNER", "MEMBER", "COLLABORATOR"]); /** - * Effective repository permissions that mark an author as internal. A user who - * can push to the repository (directly, or via a team/org grant that - * `author_association` does not surface) is a trusted maintainer, not an - * external contributor. The legacy REST `permission` field collapses the - * `maintain` role to `write`, so `admin`/`write` covers every push-capable - * role. + * Effective repository permissions that mark an author as internal: anyone + * who can push, directly or via a team/org grant `author_association` won't + * surface. The legacy REST `permission` field collapses `maintain` into + * `write`, so `admin`/`write` covers every push-capable role. * - * Exported for `resolve.ts`, which requires the same write-permission bar to - * authorize a `/ai-review` command. + * Exported for `resolve.ts`, which uses the same bar to authorize `/ai-review`. */ export const WRITE_PERMISSIONS = new Set(["admin", "write"]); @@ -147,9 +136,8 @@ export function evaluateGate(input: GateInput): GateResult { return { pass: true, reason: "internal" }; } - // Only issues in THIS repository count. A cross-repository closing keyword - // (e.g. `Closes attacker/repo#1`) links an issue the contributor controls, - // so it must never satisfy the gate. + // Only issues in this repository count; a cross-repository closing keyword + // (e.g. `Closes attacker/repo#1`) links an issue the contributor controls. const repo = input.repository.toLowerCase(); const repoIssues = input.linkedIssues.filter((issue) => issue.repository.toLowerCase() === repo); @@ -234,8 +222,6 @@ export async function evaluateAllOpenPrs(io: GateIo, repository: string): Promis return entries; } -// --- GitHub I/O (only runs when executed directly) --- - interface GraphQLIssueNode { number: number; state: "OPEN" | "CLOSED"; @@ -358,16 +344,11 @@ async function fetchOpenPullRequests( } /** - * Resolve an author's effective permission on the repository. Reflects access - * granted directly or through a team/org membership, so it recognises private - * org members that `author_association` reports only as `CONTRIBUTOR`. This - * endpoint needs just `Metadata: read` (covered by the workflow's - * `contents: read`). - * - * A 404 means the author is not a collaborator at all — the common case for - * external fork contributors, who are exactly who the gate targets — so it maps - * to `undefined` (external) rather than throwing. Other failures still throw so - * a transient API error aborts the run without wrongly closing PRs. + * Resolves an author's effective permission on the repository, recognizing + * private org members that `author_association` reports only as + * `CONTRIBUTOR`. A 404 means the author isn't a collaborator at all — the + * common case for external fork contributors — so it maps to `undefined` + * rather than throwing; other failures still throw. */ export async function fetchAuthorPermission( token: string, @@ -426,9 +407,8 @@ async function runSinglePr( const authorLogin = process.env.PR_AUTHOR_LOGIN ?? ""; const isBot = (process.env.PR_AUTHOR_TYPE ?? "User") === "Bot"; - // Resolve maintainer status from the effective repository permission unless a - // cheaper signal already settles it. `author_association` only exposes public - // org membership, so a private member must be confirmed via permission. + // `author_association` only exposes public org membership, so a private + // member must be confirmed via the effective repository permission. let permission: string | undefined; let isInternal = isBot || INTERNAL_ASSOCIATIONS.has(authorAssociation); if (!isInternal) { diff --git a/.github/scripts/mirror-slim-image.test.ts b/.github/scripts/mirror-slim-image.test.ts new file mode 100644 index 0000000000..af26acc637 --- /dev/null +++ b/.github/scripts/mirror-slim-image.test.ts @@ -0,0 +1,700 @@ +import { describe, expect, test } from "bun:test"; + +import { + copyImage, + copyNatives, + ensureEcrPublicRepo, + fetchNatives, + main, + nativesFromEvent, + uploadNativesS3, + verifyDigest, + type CommandResult, + type RunCommand, +} from "./mirror-slim-image.ts"; +import { + InvalidPayloadError, + checksumFor, + digestReference, + nativeFileNames, + nativeObjectUrl, + nativeTagPattern, + nativeTripletDigests, + parseNatives, + validateMirrorDispatch, +} from "./slim-mirror-payload.ts"; + +const DIGEST = `sha256:${"a".repeat(64)}`; +const OTHER = `sha256:${"b".repeat(64)}`; + +const ok = (stdout = ""): CommandResult => ({ ok: true, stdout, stderr: "" }); +const fail = (stderr = "", stdout = ""): CommandResult => ({ ok: false, stdout, stderr }); + +describe("validateMirrorDispatch", () => { + test("derives source and destination on workflow_dispatch", () => { + expect( + validateMirrorDispatch({ + eventName: "workflow_dispatch", + service: "postgrest", + version: "v16.2", + digest: DIGEST, + payloadSource: undefined, + payloadDestination: undefined, + }), + ).toEqual({ + service: "postgrest", + version: "v16.2", + digest: DIGEST, + source: "ghcr.io/supabase/cli/postgrest:v16.2", + destination: "public.ecr.aws/supabase/cli/postgrest:v16.2", + }); + }); + + test("requires payload URLs to match derived refs on repository_dispatch", () => { + expect(() => + validateMirrorDispatch({ + eventName: "repository_dispatch", + service: "postgrest", + version: "v16.2", + digest: DIGEST, + payloadSource: "ghcr.io/evil/cli/postgrest:v16.2", + payloadDestination: "public.ecr.aws/supabase/cli/postgrest:v16.2", + }), + ).toThrow(InvalidPayloadError); + }); + + test("rejects a digest that is not sha256", () => { + expect(() => + validateMirrorDispatch({ + eventName: "workflow_dispatch", + service: "postgrest", + version: "v16.2", + digest: "sha256:nope", + payloadSource: undefined, + payloadDestination: undefined, + }), + ).toThrow(InvalidPayloadError); + }); +}); + +describe("parseNatives", () => { + test("accepts an empty or missing list", () => { + expect(parseNatives(undefined, "v16.2")).toEqual([]); + expect(parseNatives(null, "v16.2")).toEqual([]); + expect(parseNatives([], "v16.2")).toEqual([]); + }); + + test("rejects a platform image tag", () => { + expect(() => parseNatives([{ tag: "v16.2-linux-arm64", digest: DIGEST }], "v16.2")).toThrow( + InvalidPayloadError, + ); + }); + + test("keeps a matching native tag", () => { + expect(parseNatives([{ tag: "v16.2-native-linux-arm64", digest: DIGEST }], "v16.2")).toEqual([ + { tag: "v16.2-native-linux-arm64", digest: DIGEST }, + ]); + expect(nativeTagPattern("v16.2").test("v16.2-native-darwin-arm64")).toBe(true); + }); +}); + +describe("verifyDigest", () => { + test("accepts a matching head", async () => { + const run: RunCommand = async () => ok(`${DIGEST}\n`); + await verifyDigest({ + reference: "ghcr.io/supabase/cli/postgrest:v16.2", + digest: DIGEST, + run, + log: () => undefined, + }); + }); + + test("rejects a mismatch", async () => { + const run: RunCommand = async () => ok(`${OTHER}\n`); + await expect( + verifyDigest({ + reference: "ghcr.io/supabase/cli/postgrest:v16.2", + digest: DIGEST, + run, + log: () => undefined, + }), + ).rejects.toThrow(InvalidPayloadError); + }); + + test("re-reads a stale tag until it converges", async () => { + const heads = [fail("not found [http 404]"), ok(`${OTHER}\n`), ok(`${DIGEST}\n`)]; + let reads = 0; + const run: RunCommand = async () => heads[reads++] ?? fail("unexpected read"); + await verifyDigest({ + reference: "public.ecr.aws/supabase/cli/postgrest:v16.2", + digest: DIGEST, + run, + attempts: 5, + sleep: async () => undefined, + log: () => undefined, + }); + expect(reads).toBe(3); + }); + + test("reports the last mismatch once attempts are exhausted", async () => { + let reads = 0; + const run: RunCommand = async () => { + reads++; + return ok(`${OTHER}\n`); + }; + await expect( + verifyDigest({ + reference: "public.ecr.aws/supabase/cli/postgrest:v16.2", + digest: DIGEST, + run, + attempts: 3, + sleep: async () => undefined, + log: () => undefined, + }), + ).rejects.toThrow(`resolves to ${OTHER}, expected ${DIGEST}`); + expect(reads).toBe(3); + }); + + test("fails on the first read when the head errors for another reason", async () => { + let reads = 0; + const run: RunCommand = async () => { + reads++; + return fail("unauthorized [http 401]"); + }; + await expect( + verifyDigest({ + reference: "public.ecr.aws/supabase/cli/postgrest:v16.2", + digest: DIGEST, + run, + attempts: 5, + sleep: async () => undefined, + log: () => undefined, + }), + ).rejects.toThrow("unauthorized [http 401]"); + expect(reads).toBe(1); + }); + + test("rejects a non-integer attempt count before reading", async () => { + let reads = 0; + const run: RunCommand = async () => { + reads++; + return ok(`${OTHER}\n`); + }; + await expect( + verifyDigest({ + reference: "public.ecr.aws/supabase/cli/postgrest:v16.2", + digest: DIGEST, + run, + attempts: Number.NaN, + log: () => undefined, + }), + ).rejects.toThrow(InvalidPayloadError); + expect(reads).toBe(0); + }); +}); + +describe("ensureEcrPublicRepo", () => { + test("no-ops when the repository exists", async () => { + const run: RunCommand = async () => ok(); + const logs: string[] = []; + await ensureEcrPublicRepo({ + service: "postgrest", + run, + log: (message) => logs.push(message), + }); + expect(logs).toEqual(["ECR Public repository cli/postgrest exists"]); + }); + + test("creates a missing repository", async () => { + const run: RunCommand = async (argv) => + argv.includes("describe-repositories") ? fail("not found") : ok(); + const logs: string[] = []; + await ensureEcrPublicRepo({ + service: "postgrest", + run, + log: (message) => logs.push(message), + }); + expect(logs).toEqual(["created ECR Public repository cli/postgrest"]); + }); + + test("treats a create race as success", async () => { + const run: RunCommand = async () => fail("RepositoryAlreadyExistsException"); + const logs: string[] = []; + await ensureEcrPublicRepo({ + service: "auth", + run, + log: (message) => logs.push(message), + }); + expect(logs).toEqual(["ECR Public repository cli/auth was created concurrently"]); + }); + + test("fails when create is denied", async () => { + const run: RunCommand = async () => fail("AccessDenied"); + await expect( + ensureEcrPublicRepo({ service: "postgrest", run, log: () => undefined }), + ).rejects.toThrow(/CreateRepository/); + }); +}); + +describe("copyImage", () => { + test("copies by digest with referrers", async () => { + const calls: string[][] = []; + const run: RunCommand = async (argv) => { + calls.push([...argv]); + return ok(); + }; + await copyImage({ + source: "ghcr.io/supabase/cli/postgrest:v16.2", + destination: "public.ecr.aws/supabase/cli/postgrest:v16.2", + digest: DIGEST, + run, + }); + expect(calls).toEqual([ + [ + "regctl", + "image", + "copy", + "--referrers", + "--digest-tags", + digestReference("ghcr.io/supabase/cli/postgrest:v16.2", DIGEST), + "public.ecr.aws/supabase/cli/postgrest:v16.2", + ], + ]); + }); +}); + +describe("copyNatives", () => { + test("copies each matching source and continues after a missing tag", async () => { + const calls: string[][] = []; + const run: RunCommand = async (argv) => { + calls.push([...argv]); + if (argv[1] === "manifest" && String(argv[3]).includes("linux-amd64")) return fail("missing"); + if (argv[1] === "manifest") return ok(`${DIGEST}\n`); + return ok(); + }; + const warnings: string[] = []; + const failed = await copyNatives({ + service: "postgrest", + natives: [ + { tag: "v16.2-native-linux-arm64", digest: DIGEST }, + { tag: "v16.2-native-linux-amd64", digest: DIGEST }, + ], + run, + log: (message) => { + warnings.push(message); + }, + }); + expect(failed).toBe(1); + expect(calls).toEqual([ + ["regctl", "manifest", "head", "ghcr.io/supabase/cli/postgrest:v16.2-native-linux-arm64"], + [ + "regctl", + "image", + "copy", + `ghcr.io/supabase/cli/postgrest@${DIGEST}`, + "public.ecr.aws/supabase/cli/postgrest:v16.2-native-linux-arm64", + ], + ["regctl", "manifest", "head", "ghcr.io/supabase/cli/postgrest:v16.2-native-linux-amd64"], + ]); + expect(warnings.some((line) => line.includes("linux-amd64 is missing"))).toBe(true); + }); + + test("skips a digest mismatch without copying", async () => { + const calls: string[][] = []; + const run: RunCommand = async (argv) => { + calls.push([...argv]); + return argv[1] === "manifest" ? ok(`${OTHER}\n`) : fail("should not copy"); + }; + const failed = await copyNatives({ + service: "postgrest", + natives: [{ tag: "v16.2-native-linux-arm64", digest: DIGEST }], + run, + log: () => undefined, + }); + expect(failed).toBe(1); + expect(calls).toEqual([ + ["regctl", "manifest", "head", "ghcr.io/supabase/cli/postgrest:v16.2-native-linux-arm64"], + ]); + }); +}); + +describe("main", () => { + test("validate writes GitHub outputs", async () => { + const fields: Record = {}; + const code = await main(["validate"], { + env: { + EVENT_NAME: "workflow_dispatch", + SERVICE: "postgrest", + VERSION: "v16.2", + DIGEST, + }, + run: async () => fail("unused"), + writeOutput: (next) => { + Object.assign(fields, next); + }, + }); + expect(code).toBe(0); + expect(fields["destination"]).toBe("public.ecr.aws/supabase/cli/postgrest:v16.2"); + }); + + test("verify-digest re-reads up to ATTEMPTS times", async () => { + const heads = [ok(`${OTHER}\n`), ok(`${DIGEST}\n`)]; + let reads = 0; + const code = await main(["verify-digest"], { + env: { + REFERENCE: "public.ecr.aws/supabase/cli/postgrest:v16.2", + DIGEST, + ATTEMPTS: "2", + }, + run: async () => heads[reads++] ?? fail("unexpected read"), + sleep: async () => undefined, + log: () => undefined, + }); + expect(code).toBe(0); + expect(reads).toBe(2); + }); + + test("copy-natives no-ops without a dispatch payload", async () => { + const logs: string[] = []; + const code = await main(["copy-natives"], { + env: { EVENT_NAME: "workflow_dispatch", SERVICE: "postgrest", VERSION: "v16.2" }, + run: async () => fail("unused"), + log: (message) => logs.push(message), + }); + expect(code).toBe(0); + expect(logs).toEqual(["no native artifacts in payload"]); + }); + + test("copy-natives reads natives from the dispatch event", async () => { + const natives = await nativesFromEvent( + { EVENT_NAME: "repository_dispatch", GITHUB_EVENT_PATH: "event.json" }, + "v16.2", + async () => + JSON.stringify({ + client_payload: { natives: [{ tag: "v16.2-native-linux-arm64", digest: DIGEST }] }, + }), + ); + expect(natives).toEqual([{ tag: "v16.2-native-linux-arm64", digest: DIGEST }]); + }); +}); + +const ARCHIVE_BLOB = `sha256:${"1".repeat(64)}`; +const MANIFEST_BLOB = `sha256:${"2".repeat(64)}`; +const CHECKSUM_BLOB = `sha256:${"3".repeat(64)}`; +const ARCHIVE_SHA = "d".repeat(64); + +const ociManifest = JSON.stringify({ + layers: [ + { mediaType: "application/vnd.supabase.slim.archive.v1.tar+zstd", digest: ARCHIVE_BLOB }, + { mediaType: "application/vnd.supabase.slim.manifest.v1+json", digest: MANIFEST_BLOB }, + { mediaType: "application/vnd.supabase.slim.checksum.v1", digest: CHECKSUM_BLOB }, + ], +}); + +describe("native triplet helpers", () => { + test("keeps release asset names as object keys", () => { + expect(nativeFileNames("postgrest", "v16.2", "linux-arm64")).toEqual({ + archive: "postgrest-v16.2-linux-arm64.tar.zst", + manifest: "postgrest-v16.2-linux-arm64.manifest.json", + checksum: "postgrest-v16.2-linux-arm64.SHA256SUMS", + }); + expect(nativeObjectUrl("postgrest", "v16.2", "postgrest-v16.2-linux-arm64.tar.zst")).toBe( + "https://supabase-cli-artifacts.s3.us-east-1.amazonaws.com/postgrest/v16.2/postgrest-v16.2-linux-arm64.tar.zst", + ); + }); + + test("selects triplet blobs by media type and rejects incomplete artifacts", () => { + expect(nativeTripletDigests(ociManifest)).toEqual({ + archive: ARCHIVE_BLOB, + manifest: MANIFEST_BLOB, + checksum: CHECKSUM_BLOB, + }); + expect( + nativeTripletDigests(JSON.stringify({ layers: [{ mediaType: "x", digest: ARCHIVE_BLOB }] })), + ).toBeUndefined(); + }); + + test("matches sums lines by archive name", () => { + const sums = `${ARCHIVE_SHA.toUpperCase()} postgrest-v16.2-linux-arm64.tar.zst\n`; + expect(checksumFor(sums, "postgrest-v16.2-linux-arm64.tar.zst")).toBe(ARCHIVE_SHA); + expect(checksumFor(sums, "postgrest-v16.2-linux-amd64.tar.zst")).toBeUndefined(); + }); +}); + +describe("fetchNatives", () => { + const io = ( + overrides: { readonly sha?: string; readonly manifest?: string; readonly head?: string } = {}, + ) => { + const files: Record = {}; + const calls: string[][] = []; + const run: RunCommand = async (argv) => { + calls.push([...argv]); + if (argv[1] === "manifest" && argv[2] === "head") return ok(`${overrides.head ?? DIGEST}\n`); + return argv[1] === "manifest" ? ok(ociManifest) : fail("unexpected"); + }; + return { + calls, + run, + runToFile: async (argv: ReadonlyArray, outputPath: string) => { + calls.push([...argv, `> ${outputPath}`]); + const digest = argv[4]; + files[outputPath] = + digest === MANIFEST_BLOB + ? (overrides.manifest ?? + JSON.stringify({ service: "postgrest", version: "v16.2", target: "linux-arm64" })) + : digest === CHECKSUM_BLOB + ? `${ARCHIVE_SHA} postgrest-v16.2-linux-arm64.tar.zst\n` + : "archive-bytes"; + return ok(); + }, + readText: async (path: string) => files[path] ?? "", + sha256File: async () => overrides.sha ?? ARCHIVE_SHA, + mkdir: () => undefined, + log: () => undefined, + }; + }; + + test("downloads the triplet under release asset names and verifies it", async () => { + const deps = io(); + const fetched = await fetchNatives({ + service: "postgrest", + version: "v16.2", + natives: [{ tag: "v16.2-native-linux-arm64", digest: DIGEST }], + outputDir: "/tmp/natives", + ...deps, + }); + expect(fetched).toEqual([ + { target: "linux-arm64", files: nativeFileNames("postgrest", "v16.2", "linux-arm64") }, + ]); + expect(deps.calls[0]).toEqual([ + "regctl", + "manifest", + "head", + "ghcr.io/supabase/cli/postgrest:v16.2-native-linux-arm64", + ]); + expect(deps.calls[1]).toEqual([ + "regctl", + "manifest", + "get", + `ghcr.io/supabase/cli/postgrest@${DIGEST}`, + "--format", + "raw-body", + ]); + expect(deps.calls[2]).toEqual([ + "regctl", + "blob", + "get", + "ghcr.io/supabase/cli/postgrest", + ARCHIVE_BLOB, + "> /tmp/natives/linux-arm64/postgrest-v16.2-linux-arm64.tar.zst", + ]); + }); + + test("skips a native whose tag no longer resolves to the dispatched digest", async () => { + const deps = io({ head: OTHER }); + const fetched = await fetchNatives({ + service: "postgrest", + version: "v16.2", + natives: [{ tag: "v16.2-native-linux-arm64", digest: DIGEST }], + outputDir: "/tmp/natives", + ...deps, + }); + expect(fetched).toEqual([]); + expect(deps.calls).toHaveLength(1); + }); + + test("keeps the first entry for a repeated target and never refetches it", async () => { + const deps = io(); + const fetched = await fetchNatives({ + service: "postgrest", + version: "v16.2", + natives: [ + { tag: "v16.2-native-linux-arm64", digest: DIGEST }, + { tag: "v16.2-native-linux-arm64", digest: OTHER }, + ], + outputDir: "/tmp/natives", + ...deps, + }); + expect(fetched.map((item) => item.target)).toEqual(["linux-arm64"]); + expect(deps.calls.filter((argv) => argv[1] === "manifest" && argv[2] === "head")).toHaveLength( + 1, + ); + }); + + test("drops a target whose manifest is not JSON", async () => { + const fetched = await fetchNatives({ + service: "postgrest", + version: "v16.2", + natives: [{ tag: "v16.2-native-linux-arm64", digest: DIGEST }], + outputDir: "/tmp/natives", + ...io({ manifest: "{not json" }), + }); + expect(fetched).toEqual([]); + expect(nativeTripletDigests("")).toBeUndefined(); + }); + + test("drops a target whose archive does not match its sums file", async () => { + const fetched = await fetchNatives({ + service: "postgrest", + version: "v16.2", + natives: [{ tag: "v16.2-native-linux-arm64", digest: DIGEST }], + outputDir: "/tmp/natives", + ...io({ sha: "e".repeat(64) }), + }); + expect(fetched).toEqual([]); + }); + + test("drops a target whose manifest names another release", async () => { + const fetched = await fetchNatives({ + service: "postgrest", + version: "v16.2", + natives: [{ tag: "v16.2-native-linux-arm64", digest: DIGEST }], + outputDir: "/tmp/natives", + ...io({ + manifest: JSON.stringify({ service: "postgrest", version: "v16.1", target: "linux-arm64" }), + }), + }); + expect(fetched).toEqual([]); + }); +}); + +describe("uploadNativesS3", () => { + const fetched = [ + { target: "linux-arm64", files: nativeFileNames("postgrest", "v16.2", "linux-arm64") }, + ]; + + test("copies archive, manifest, then sums and verifies anonymous reads", async () => { + const calls: string[][] = []; + const checked: string[] = []; + await uploadNativesS3({ + service: "postgrest", + version: "v16.2", + inputDir: "/tmp/natives", + fetched, + run: async (argv) => { + calls.push([...argv]); + return ok(); + }, + httpStatus: async (url, method) => { + checked.push(`${method} ${url}`); + return url.endsWith("/") ? 403 : 200; + }, + log: () => undefined, + }); + expect(calls.map((argv) => argv.at(-1))).toEqual([ + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-linux-arm64.tar.zst", + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-linux-arm64.manifest.json", + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-linux-arm64.SHA256SUMS", + ]); + expect(calls[0]).toContain("/tmp/natives/linux-arm64/postgrest-v16.2-linux-arm64.tar.zst"); + expect(calls.every((argv) => argv[1] === "s3" && argv[2] === "cp")).toBe(true); + expect(checked).toEqual([ + "HEAD https://supabase-cli-artifacts.s3.us-east-1.amazonaws.com/postgrest/v16.2/postgrest-v16.2-linux-arm64.tar.zst", + "HEAD https://supabase-cli-artifacts.s3.us-east-1.amazonaws.com/postgrest/v16.2/postgrest-v16.2-linux-arm64.manifest.json", + "HEAD https://supabase-cli-artifacts.s3.us-east-1.amazonaws.com/postgrest/v16.2/postgrest-v16.2-linux-arm64.SHA256SUMS", + "GET https://supabase-cli-artifacts.s3.us-east-1.amazonaws.com/", + ]); + }); + + test("fails when an object is not publicly readable", async () => { + await expect( + uploadNativesS3({ + service: "postgrest", + version: "v16.2", + inputDir: "/tmp/natives", + fetched, + run: async () => ok(), + httpStatus: async () => 403, + log: () => undefined, + }), + ).rejects.toThrow(/not publicly readable/); + }); + + test("fails when the bucket starts listing", async () => { + await expect( + uploadNativesS3({ + service: "postgrest", + version: "v16.2", + inputDir: "/tmp/natives", + fetched, + run: async () => ok(), + httpStatus: async () => 200, + log: () => undefined, + }), + ).rejects.toThrow(/listing must stay closed/); + }); +}); + +describe("main fetch-natives", () => { + test("writes count=0 without a dispatch payload", async () => { + const fields: Record = {}; + const code = await main(["fetch-natives"], { + env: { + EVENT_NAME: "workflow_dispatch", + SERVICE: "postgrest", + VERSION: "v16.2", + OUTPUT_DIR: "/tmp/natives", + }, + run: async () => fail("unused"), + log: () => undefined, + writeOutput: (next) => { + Object.assign(fields, next); + }, + }); + expect(code).toBe(0); + expect(fields).toEqual({ count: "0", targets: "" }); + }); + + test("exits 1 when the payload names natives but none can be verified", async () => { + const fields: Record = {}; + const event = JSON.stringify({ + client_payload: { natives: [{ tag: "v16.2-native-linux-arm64", digest: DIGEST }] }, + }); + const code = await main(["fetch-natives"], { + env: { + EVENT_NAME: "repository_dispatch", + GITHUB_EVENT_PATH: "/tmp/event.json", + SERVICE: "postgrest", + VERSION: "v16.2", + OUTPUT_DIR: "/tmp/natives", + }, + run: async () => fail("manifest unknown"), + readText: async () => event, + log: () => undefined, + writeOutput: (next) => { + Object.assign(fields, next); + }, + }); + expect(code).toBe(1); + expect(fields).toEqual({ count: "0", targets: "" }); + }); +}); + +describe("main upload-natives-s3", () => { + test("uploads every target listed in TARGETS", async () => { + const uploaded: string[] = []; + const code = await main(["upload-natives-s3"], { + env: { + SERVICE: "postgrest", + VERSION: "v16.2", + INPUT_DIR: "/tmp/natives", + TARGETS: "linux-arm64, darwin-arm64", + }, + run: async (argv) => { + uploaded.push(String(argv.at(-1))); + return ok(); + }, + httpStatus: async (url) => (url.endsWith("/") ? 403 : 200), + log: () => undefined, + }); + expect(code).toBe(0); + expect(uploaded).toEqual([ + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-linux-arm64.tar.zst", + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-linux-arm64.manifest.json", + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-linux-arm64.SHA256SUMS", + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-darwin-arm64.tar.zst", + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-darwin-arm64.manifest.json", + "s3://supabase-cli-artifacts/postgrest/v16.2/postgrest-v16.2-darwin-arm64.SHA256SUMS", + ]); + }); +}); diff --git a/.github/scripts/mirror-slim-image.ts b/.github/scripts/mirror-slim-image.ts new file mode 100644 index 0000000000..6e0b0abc7f --- /dev/null +++ b/.github/scripts/mirror-slim-image.ts @@ -0,0 +1,597 @@ +/** + * GHCR → ECR Public copy for slim images and optional native OCI tags, plus the + * GHCR → S3 copy of the native triplets. Registry logins and AWS credentials stay + * in the workflow; this script is the replayable logic. + * + * bun .github/scripts/mirror-slim-image.ts validate + * bun .github/scripts/mirror-slim-image.ts verify-digest + * bun .github/scripts/mirror-slim-image.ts ensure-repo + * bun .github/scripts/mirror-slim-image.ts copy-image + * bun .github/scripts/mirror-slim-image.ts copy-natives + * bun .github/scripts/mirror-slim-image.ts fetch-natives + * bun .github/scripts/mirror-slim-image.ts upload-natives-s3 + */ + +import { appendFileSync, mkdirSync } from "node:fs"; +import { join } from "node:path"; + +import { + DEST_REGISTRY, + InvalidPayloadError, + S3_BASE_URL, + S3_BUCKET, + SOURCE_REGISTRY, + checksumFor, + digestReference, + manifestMatches, + nativeFileNames, + nativeObjectKey, + nativeObjectUrl, + nativeTargetOf, + nativeTripletDigests, + parseNatives, + validateMirrorDispatch, + type NativeArtifact, + type NativeFiles, +} from "./slim-mirror-payload.ts"; + +export type CommandResult = { + readonly ok: boolean; + readonly stdout: string; + readonly stderr: string; +}; + +export type RunCommand = (argv: ReadonlyArray) => Promise; + +/** Like {@link RunCommand} but streams stdout into `outputPath`, for binary blobs. */ +export type RunCommandToFile = ( + argv: ReadonlyArray, + outputPath: string, +) => Promise; + +export type MirrorIo = { + readonly env: NodeJS.Dict; + readonly run: RunCommand; + readonly runToFile?: RunCommandToFile; + readonly log?: (message: string) => void; + readonly sleep?: (ms: number) => Promise; + readonly readText?: (path: string) => Promise; + readonly sha256File?: (path: string) => Promise; + readonly httpStatus?: (url: string, method: "HEAD" | "GET") => Promise; + readonly mkdir?: (path: string) => void; + readonly writeOutput?: (fields: Readonly>) => void; +}; + +const usage = `Usage: + bun .github/scripts/mirror-slim-image.ts validate + bun .github/scripts/mirror-slim-image.ts verify-digest + bun .github/scripts/mirror-slim-image.ts ensure-repo + bun .github/scripts/mirror-slim-image.ts copy-image + bun .github/scripts/mirror-slim-image.ts copy-natives + bun .github/scripts/mirror-slim-image.ts fetch-natives + bun .github/scripts/mirror-slim-image.ts upload-natives-s3`; + +const envValue = (env: NodeJS.Dict, key: string): string => env[key] ?? ""; + +const requireEnv = (env: NodeJS.Dict, key: string): string => { + const value = envValue(env, key).trim(); + if (value === "") throw new InvalidPayloadError(`missing required environment variable: ${key}`); + return value; +}; + +const writeGithubOutput = ( + env: NodeJS.Dict, + fields: Readonly>, + writeOutput?: MirrorIo["writeOutput"], +): void => { + if (writeOutput !== undefined) { + writeOutput(fields); + return; + } + const body = `${Object.entries(fields) + .map(([key, value]) => `${key}=${value}`) + .join("\n")}\n`; + const path = env["GITHUB_OUTPUT"]; + if (path !== undefined && path.trim() !== "") appendFileSync(path, body); + else process.stdout.write(body); +}; + +const VERIFY_RETRY_DELAY_MS = 10_000; + +const NOT_FOUND = /\bnot found\b|\bhttp 404\b/i; + +/** + * Checks that `reference` resolves to `digest`, re-reading a mismatch or 404 up to + * `attempts` times. ECR Public can keep serving a tag's previous digest (or a 404 for a + * new tag) for a short while after a push, so the destination check needs more than one read. + */ +export const verifyDigest = async (options: { + readonly reference: string; + readonly digest: string; + readonly run: RunCommand; + readonly attempts?: number; + readonly sleep?: (ms: number) => Promise; + readonly log?: (message: string) => void; +}): Promise => { + const log = options.log ?? console.log; + const attempts = options.attempts ?? 1; + if (!Number.isInteger(attempts) || attempts < 1) + throw new InvalidPayloadError(`attempts must be a positive integer, got ${attempts}`); + const sleep = options.sleep ?? Bun.sleep; + for (let attempt = 1; ; attempt++) { + const head = await options.run(["regctl", "manifest", "head", options.reference]); + const live = head.stdout.trim(); + if (head.ok && live === options.digest) { + log(`${options.reference} resolves to ${options.digest}`); + return; + } + const resolved = head.ok ? live : live || "missing"; + const detail = !head.ok && head.stderr.trim() !== "" ? `: ${head.stderr.trim()}` : ""; + const mismatch = `${options.reference} resolves to ${resolved}, expected ${options.digest}${detail}`; + const stale = head.ok || NOT_FOUND.test(head.stderr); + if (!stale || attempt >= attempts) throw new InvalidPayloadError(mismatch); + log(`${mismatch}; re-reading (attempt ${attempt + 1}/${attempts})`); + await sleep(VERIFY_RETRY_DELAY_MS); + } +}; + +export const ensureEcrPublicRepo = async (options: { + readonly service: string; + readonly run: RunCommand; + readonly log?: (message: string) => void; +}): Promise => { + const log = options.log ?? console.log; + const repoName = `cli/${options.service}`; + const described = await options.run([ + "aws", + "ecr-public", + "describe-repositories", + "--repository-names", + repoName, + "--region", + "us-east-1", + ]); + if (described.ok) { + log(`ECR Public repository ${repoName} exists`); + return; + } + const created = await options.run([ + "aws", + "ecr-public", + "create-repository", + "--repository-name", + repoName, + "--region", + "us-east-1", + ]); + if (created.ok) { + log(`created ECR Public repository ${repoName}`); + return; + } + const detail = `${created.stdout}\n${created.stderr}`; + if (detail.includes("RepositoryAlreadyExistsException")) { + log(`ECR Public repository ${repoName} was created concurrently`); + return; + } + if (detail.trim() !== "") log(detail.trim()); + throw new InvalidPayloadError( + `ECR Public repository '${repoName}' does not exist and this role cannot create it (missing ecr-public:CreateRepository). Create it once manually — aws ecr-public create-repository --repository-name '${repoName}' --region us-east-1 — then re-run this workflow.`, + ); +}; + +export const copyImage = async (options: { + readonly source: string; + readonly destination: string; + readonly digest: string; + readonly run: RunCommand; +}): Promise => { + const copied = await options.run([ + "regctl", + "image", + "copy", + "--referrers", + "--digest-tags", + digestReference(options.source, options.digest), + options.destination, + ]); + if (!copied.ok) + throw new InvalidPayloadError(copied.stderr.trim() || `copy failed: ${options.source}`); +}; + +export const copyNatives = async (options: { + readonly service: string; + readonly natives: ReadonlyArray; + readonly run: RunCommand; + readonly log?: (message: string) => void; +}): Promise => { + const log = options.log ?? console.log; + let failed = 0; + for (const { tag, digest } of options.natives) { + const source = `${SOURCE_REGISTRY}/${options.service}:${tag}`; + const destination = `${DEST_REGISTRY}/${options.service}:${tag}`; + const head = await options.run(["regctl", "manifest", "head", source]); + const sourceDigest = head.stdout.trim(); + if (!head.ok) { + log(`::warning::native source ${source} is missing`); + if (head.stderr.trim() !== "") log(head.stderr.trim()); + failed += 1; + continue; + } + if (sourceDigest !== digest) { + log(`::warning::native source ${source} resolves to ${sourceDigest}, expected ${digest}`); + failed += 1; + continue; + } + const copied = await options.run([ + "regctl", + "image", + "copy", + digestReference(source, digest), + destination, + ]); + if (!copied.ok) { + log(`::warning::native copy failed: ${source} -> ${destination}`); + if (copied.stderr.trim() !== "") log(copied.stderr.trim()); + failed += 1; + continue; + } + log(`mirrored ${destination}@${digest}`); + } + return failed; +}; + +export type FetchedNative = { + readonly target: string; + readonly files: NativeFiles; +}; + +/** + * Pulls each native triplet from GHCR by digest into `//` under its + * release asset names, then checks the archive against its own SHA256SUMS and the manifest + * against the dispatch fields. Returns the targets that passed; failures are warnings. + */ +export const fetchNatives = async (options: { + readonly service: string; + readonly version: string; + readonly natives: ReadonlyArray; + readonly outputDir: string; + readonly run: RunCommand; + readonly runToFile: RunCommandToFile; + readonly readText: (path: string) => Promise; + readonly sha256File: (path: string) => Promise; + readonly mkdir: (path: string) => void; + readonly log?: (message: string) => void; +}): Promise> => { + const log = options.log ?? console.log; + const repository = `${SOURCE_REGISTRY}/${options.service}`; + const fetched: FetchedNative[] = []; + const seen = new Set(); + for (const { tag, digest } of options.natives) { + const target = nativeTargetOf(tag, options.version); + if (target === undefined) { + log(`::warning::native tag ${tag} does not name a target`); + continue; + } + // Each target stages into one directory, so a second entry could overwrite verified files. + if (seen.has(target)) { + log(`::warning::native tag ${tag} repeats target ${target}; keeping the first entry`); + continue; + } + seen.add(target); + // The digest is untrusted dispatch data: only publish what the tag currently points to. + const source = `${repository}:${tag}`; + const head = await options.run(["regctl", "manifest", "head", source]); + if (!head.ok) { + log(`::warning::native source ${source} is missing`); + if (head.stderr.trim() !== "") log(head.stderr.trim()); + continue; + } + if (head.stdout.trim() !== digest) { + log( + `::warning::native source ${source} resolves to ${head.stdout.trim()}, expected ${digest}`, + ); + continue; + } + const reference = `${repository}@${digest}`; + const manifest = await options.run([ + "regctl", + "manifest", + "get", + reference, + "--format", + "raw-body", + ]); + if (!manifest.ok) { + log(`::warning::native manifest ${reference} is missing`); + if (manifest.stderr.trim() !== "") log(manifest.stderr.trim()); + continue; + } + const digests = nativeTripletDigests(manifest.stdout); + if (digests === undefined) { + log(`::warning::native ${reference} is not a complete archive/manifest/checksum triplet`); + continue; + } + const files = nativeFileNames(options.service, options.version, target); + const dir = join(options.outputDir, target); + options.mkdir(dir); + let complete = true; + for (const part of ["archive", "manifest", "checksum"] as const) { + const blob = await options.runToFile( + ["regctl", "blob", "get", repository, digests[part]], + join(dir, files[part]), + ); + if (!blob.ok) { + log(`::warning::native blob ${digests[part]} for ${reference} failed to download`); + if (blob.stderr.trim() !== "") log(blob.stderr.trim()); + complete = false; + break; + } + } + if (!complete) continue; + const expected = checksumFor(await options.readText(join(dir, files.checksum)), files.archive); + if (expected === undefined) { + log(`::warning::${files.checksum} has no entry for ${files.archive}`); + continue; + } + const actual = (await options.sha256File(join(dir, files.archive))).toLowerCase(); + if (actual !== expected) { + log(`::warning::${files.archive} hashes to ${actual}, ${files.checksum} says ${expected}`); + continue; + } + const matches = manifestMatches(await options.readText(join(dir, files.manifest)), { + service: options.service, + version: options.version, + target, + }); + if (!matches) { + log( + `::warning::${files.manifest} does not describe ${options.service} ${options.version} ${target}`, + ); + continue; + } + log(`verified ${files.archive} from ${reference}`); + fetched.push({ target, files }); + } + return fetched; +}; + +const CONTENT_TYPES: Readonly> = { + archive: "application/zstd", + manifest: "application/json", + checksum: "text/plain", +}; + +/** + * Copies verified triplets to `s3://///` with plain `aws s3 cp` + * (the publisher role has PutObject only, so no `sync`), checksum last so a reader never + * pairs a new sums file with an older archive for longer than the upload takes. Then checks + * every object anonymously and that the bucket root still refuses listing. + */ +export const uploadNativesS3 = async (options: { + readonly service: string; + readonly version: string; + readonly inputDir: string; + readonly fetched: ReadonlyArray; + readonly run: RunCommand; + readonly httpStatus: (url: string, method: "HEAD" | "GET") => Promise; + readonly log?: (message: string) => void; +}): Promise => { + const log = options.log ?? console.log; + const uploaded: string[] = []; + for (const { target, files } of options.fetched) { + for (const part of ["archive", "manifest", "checksum"] as const) { + const key = nativeObjectKey(options.service, options.version, files[part]); + const copied = await options.run([ + "aws", + "s3", + "cp", + "--only-show-errors", + "--content-type", + CONTENT_TYPES[part], + join(options.inputDir, target, files[part]), + `s3://${S3_BUCKET}/${key}`, + ]); + if (!copied.ok) + throw new InvalidPayloadError( + copied.stderr.trim() || `upload failed: s3://${S3_BUCKET}/${key}`, + ); + uploaded.push(files[part]); + } + log(`uploaded ${target} to s3://${S3_BUCKET}/${options.service}/${options.version}/`); + } + for (const fileName of uploaded) { + const url = nativeObjectUrl(options.service, options.version, fileName); + const status = await options.httpStatus(url, "HEAD"); + if (status !== 200) + throw new InvalidPayloadError(`${url} is not publicly readable (HTTP ${status})`); + } + const listing = await options.httpStatus(`${S3_BASE_URL}/`, "GET"); + if (listing !== 403) + throw new InvalidPayloadError(`bucket listing must stay closed, got HTTP ${listing}`); + log( + `${uploaded.length} objects readable at ${S3_BASE_URL}/${options.service}/${options.version}/`, + ); +}; + +export const nativesFromEvent = async ( + env: NodeJS.Dict, + version: string, + readText: (path: string) => Promise, +): Promise> => { + if (envValue(env, "EVENT_NAME") !== "repository_dispatch") return []; + const path = envValue(env, "GITHUB_EVENT_PATH").trim(); + if (path === "") + throw new InvalidPayloadError("GITHUB_EVENT_PATH is required for repository_dispatch"); + const event: unknown = JSON.parse(await readText(path)); + const payload = + typeof event === "object" && event !== null && "client_payload" in event + ? (event as { client_payload?: { natives?: unknown } }).client_payload + : undefined; + return parseNatives(payload?.natives, version); +}; + +const defaultReadText = (path: string): Promise => Bun.file(path).text(); + +const defaultSpawn: RunCommand = async (argv) => { + const proc = Bun.spawn([...argv], { stdout: "pipe", stderr: "pipe" }); + const [stdout, stderr, exit] = await Promise.all([ + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + proc.exited, + ]); + return { ok: exit === 0, stdout, stderr }; +}; + +const defaultSpawnToFile: RunCommandToFile = async (argv, outputPath) => { + const proc = Bun.spawn([...argv], { stdout: "pipe", stderr: "pipe" }); + const [, stderr, exit] = await Promise.all([ + Bun.write(outputPath, new Response(proc.stdout)), + new Response(proc.stderr).text(), + proc.exited, + ]); + return { ok: exit === 0, stdout: "", stderr }; +}; + +const defaultSha256File = async (path: string): Promise => { + const hasher = new Bun.CryptoHasher("sha256"); + for await (const chunk of Bun.file(path).stream()) hasher.update(chunk); + return hasher.digest("hex"); +}; + +const defaultHttpStatus = async (url: string, method: "HEAD" | "GET"): Promise => + (await fetch(url, { method, redirect: "manual" })).status; + +const defaultMkdir = (path: string): void => { + mkdirSync(path, { recursive: true }); +}; + +const fetchedFromTargets = (service: string, version: string, targets: string): FetchedNative[] => + targets + .split(",") + .map((target) => target.trim()) + .filter((target) => target !== "") + .map((target) => ({ + target, + files: nativeFileNames(service, version, target), + })); + +export const main = async (argv: ReadonlyArray, io: MirrorIo): Promise => { + const command = argv[0]; + const log = io.log ?? console.log; + const readText = io.readText ?? defaultReadText; + if (command === undefined || command === "-h" || command === "--help") { + log(usage); + return command === undefined ? 2 : 0; + } + if (command === "validate") { + const refs = validateMirrorDispatch({ + eventName: envValue(io.env, "EVENT_NAME"), + service: envValue(io.env, "SERVICE"), + version: envValue(io.env, "VERSION"), + digest: envValue(io.env, "DIGEST"), + payloadSource: io.env["PAYLOAD_SOURCE"], + payloadDestination: io.env["PAYLOAD_DESTINATION"], + }); + writeGithubOutput(io.env, refs, io.writeOutput); + return 0; + } + if (command === "verify-digest") { + await verifyDigest({ + reference: requireEnv(io.env, "REFERENCE"), + digest: requireEnv(io.env, "DIGEST"), + run: io.run, + attempts: Number(envValue(io.env, "ATTEMPTS").trim() || "1"), + sleep: io.sleep, + log, + }); + return 0; + } + if (command === "ensure-repo") { + await ensureEcrPublicRepo({ + service: requireEnv(io.env, "SERVICE"), + run: io.run, + log, + }); + return 0; + } + if (command === "copy-image") { + await copyImage({ + source: requireEnv(io.env, "SOURCE"), + destination: requireEnv(io.env, "DESTINATION"), + digest: requireEnv(io.env, "DIGEST"), + run: io.run, + }); + return 0; + } + if (command === "copy-natives") { + const service = requireEnv(io.env, "SERVICE"); + const version = requireEnv(io.env, "VERSION"); + const natives = await nativesFromEvent(io.env, version, readText); + if (natives.length === 0) { + log("no native artifacts in payload"); + return 0; + } + return await copyNatives({ service, natives, run: io.run, log }); + } + if (command === "fetch-natives") { + const service = requireEnv(io.env, "SERVICE"); + const version = requireEnv(io.env, "VERSION"); + const outputDir = requireEnv(io.env, "OUTPUT_DIR"); + const natives = await nativesFromEvent(io.env, version, readText); + const fetched = + natives.length === 0 + ? [] + : await fetchNatives({ + service, + version, + natives, + outputDir, + run: io.run, + runToFile: io.runToFile ?? defaultSpawnToFile, + readText, + sha256File: io.sha256File ?? defaultSha256File, + mkdir: io.mkdir ?? defaultMkdir, + log, + }); + if (natives.length === 0) log("no native artifacts in payload"); + writeGithubOutput( + io.env, + { + count: String(fetched.length), + targets: fetched.map((item) => item.target).join(","), + }, + io.writeOutput, + ); + return natives.length > 0 && fetched.length === 0 ? 1 : 0; + } + if (command === "upload-natives-s3") { + const service = requireEnv(io.env, "SERVICE"); + const version = requireEnv(io.env, "VERSION"); + await uploadNativesS3({ + service, + version, + inputDir: requireEnv(io.env, "INPUT_DIR"), + fetched: fetchedFromTargets(service, version, requireEnv(io.env, "TARGETS")), + run: io.run, + httpStatus: io.httpStatus ?? defaultHttpStatus, + log, + }); + return 0; + } + log(usage); + return 2; +}; + +if (import.meta.main) { + try { + const code = await main(process.argv.slice(2), { + env: process.env, + run: defaultSpawn, + }); + process.exit(code); + } catch (cause) { + const message = cause instanceof Error ? cause.message : String(cause); + console.error(`::error::${message}`); + process.exit(1); + } +} diff --git a/.github/scripts/slim-mirror-payload.ts b/.github/scripts/slim-mirror-payload.ts new file mode 100644 index 0000000000..74191dfebf --- /dev/null +++ b/.github/scripts/slim-mirror-payload.ts @@ -0,0 +1,234 @@ +/** + * Untrusted `mirror-slim-image` dispatch fields. Source and destination URLs + * are derived here; payload strings are only accepted when they match. + */ + +const SERVICE_PATTERN = /^[a-z][a-z0-9-]*$/; +export const VERSION_PATTERN = /^[A-Za-z0-9._-]+$/; +export const DIGEST_PATTERN = /^sha256:[0-9a-f]{64}$/; + +export const SOURCE_REGISTRY = "ghcr.io/supabase/cli"; +export const DEST_REGISTRY = "public.ecr.aws/supabase/cli"; +const NATIVE_TARGETS = ["linux-arm64", "linux-amd64", "darwin-arm64"] as const; + +export class InvalidPayloadError extends Error {} + +const imageSource = (service: string, version: string): string => + `${SOURCE_REGISTRY}/${service}:${version}`; + +const imageDestination = (service: string, version: string): string => + `${DEST_REGISTRY}/${service}:${version}`; + +export const digestReference = (tagged: string, digest: string): string => { + const colon = tagged.lastIndexOf(":"); + if (colon <= 0) throw new InvalidPayloadError(`invalid image reference: ${tagged}`); + return `${tagged.slice(0, colon)}@${digest}`; +}; + +export const escapeRegExp = (value: string): string => value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + +export const nativeTagPattern = (version: string): RegExp => + new RegExp(`^${escapeRegExp(version)}-native-(${NATIVE_TARGETS.join("|")})$`); + +/** `service` / `version` / `digest` are the image fields. Extra keys such as `natives[]` are ignored. */ +const validatePayload = (input: { + readonly service: string; + readonly version: string; + readonly digest: string; +}): void => { + if (!SERVICE_PATTERN.test(input.service)) + throw new InvalidPayloadError(`invalid service name: '${input.service}'`); + if (!VERSION_PATTERN.test(input.version)) + throw new InvalidPayloadError(`invalid version: '${input.version}'`); + if (!DIGEST_PATTERN.test(input.digest)) + throw new InvalidPayloadError(`invalid digest: '${input.digest}'`); +}; + +export type MirrorRefs = { + readonly service: string; + readonly version: string; + readonly digest: string; + readonly source: string; + readonly destination: string; +}; + +export const validateMirrorDispatch = (input: { + readonly eventName: string; + readonly service: string; + readonly version: string; + readonly digest: string; + readonly payloadSource: string | undefined; + readonly payloadDestination: string | undefined; +}): MirrorRefs => { + validatePayload(input); + const source = imageSource(input.service, input.version); + const destination = imageDestination(input.service, input.version); + if (input.eventName === "repository_dispatch") { + if (input.payloadSource !== source) + throw new InvalidPayloadError( + `payload source '${input.payloadSource}' does not match derived '${source}'`, + ); + if (input.payloadDestination !== destination) + throw new InvalidPayloadError( + `payload destination '${input.payloadDestination}' does not match derived '${destination}'`, + ); + } + return { + service: input.service, + version: input.version, + digest: input.digest, + source, + destination, + }; +}; + +export type NativeArtifact = { + readonly tag: string; + readonly digest: string; +}; + +export const parseNatives = (raw: unknown, version: string): ReadonlyArray => { + if (raw === undefined || raw === null) return []; + if (!Array.isArray(raw)) throw new InvalidPayloadError("natives must be a JSON array"); + const tagRe = nativeTagPattern(version); + const rows: NativeArtifact[] = []; + for (const item of raw) { + if (typeof item !== "object" || item === null || !("tag" in item) || !("digest" in item)) + throw new InvalidPayloadError(`invalid native entry: ${JSON.stringify(item)}`); + const tag = (item as { tag: unknown }).tag; + const digest = (item as { digest: unknown }).digest; + if (typeof tag !== "string" || !tagRe.test(tag)) + throw new InvalidPayloadError(`invalid native tag: ${String(tag)}`); + if (typeof digest !== "string" || !DIGEST_PATTERN.test(digest)) + throw new InvalidPayloadError(`invalid native digest: ${String(digest)}`); + rows.push({ tag, digest }); + } + return rows; +}; + +export const S3_BUCKET = "supabase-cli-artifacts"; +export const S3_BASE_URL = `https://${S3_BUCKET}.s3.us-east-1.amazonaws.com`; + +const ARCHIVE_MEDIA_TYPE = "application/vnd.supabase.slim.archive.v1.tar+zstd"; +const MANIFEST_MEDIA_TYPE = "application/vnd.supabase.slim.manifest.v1+json"; +const CHECKSUM_MEDIA_TYPE = "application/vnd.supabase.slim.checksum.v1"; + +type NativeTarget = (typeof NATIVE_TARGETS)[number]; + +export const nativeTargetOf = (tag: string, version: string): NativeTarget | undefined => { + const match = nativeTagPattern(version).exec(tag); + return match === null ? undefined : (match[1] as NativeTarget); +}; + +/** Release asset stem shared by GitHub Releases, the OCI triplet, and the S3 mirror. */ +const nativeAssetName = (service: string, version: string, target: string): string => + `${service}-${version}-${target}`; + +export type NativeFiles = { + readonly archive: string; + readonly manifest: string; + readonly checksum: string; +}; + +export const nativeFileNames = (service: string, version: string, target: string): NativeFiles => { + const stem = nativeAssetName(service, version, target); + return { + archive: `${stem}.tar.zst`, + manifest: `${stem}.manifest.json`, + checksum: `${stem}.SHA256SUMS`, + }; +}; + +export const nativeObjectKey = (service: string, version: string, fileName: string): string => + `${service}/${version}/${fileName}`; + +export const nativeObjectUrl = (service: string, version: string, fileName: string): string => + `${S3_BASE_URL}/${nativeObjectKey(service, version, fileName)}`; + +type OciLayer = { + readonly mediaType: string; + readonly digest: string; + readonly title: string; +}; + +// Registry and blob contents are untrusted input; malformed JSON drops one target, not the run. +const parseJsonRecord = (raw: string): Record | undefined => { + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch { + return undefined; + } + return typeof parsed === "object" && parsed !== null + ? (parsed as Record) + : undefined; +}; + +const layersOf = (rawManifest: string): ReadonlyArray => { + const record = parseJsonRecord(rawManifest); + if (record === undefined) return []; + const raw = record["layers"] ?? record["blobs"]; + if (!Array.isArray(raw)) return []; + return raw.flatMap((entry) => { + if (typeof entry !== "object" || entry === null) return []; + const layer = entry as Record; + const digest = typeof layer["digest"] === "string" ? layer["digest"] : ""; + if (!DIGEST_PATTERN.test(digest)) return []; + const annotations = + typeof layer["annotations"] === "object" && layer["annotations"] !== null + ? (layer["annotations"] as Record) + : {}; + const title = annotations["org.opencontainers.image.title"]; + return [ + { + mediaType: typeof layer["mediaType"] === "string" ? layer["mediaType"] : "", + digest, + title: typeof title === "string" ? title : "", + }, + ]; + }); +}; + +/** Blob digests of the native triplet inside an OCI artifact manifest, or undefined when incomplete. */ +export const nativeTripletDigests = (rawManifest: string): NativeFiles | undefined => { + const layers = layersOf(rawManifest); + const archive = layers.find( + (layer) => layer.mediaType === ARCHIVE_MEDIA_TYPE || layer.title.endsWith(".tar.zst"), + ); + const manifest = layers.find( + (layer) => layer.mediaType === MANIFEST_MEDIA_TYPE || layer.title.endsWith(".manifest.json"), + ); + const checksum = layers.find( + (layer) => layer.mediaType === CHECKSUM_MEDIA_TYPE || layer.title.endsWith(".SHA256SUMS"), + ); + if (archive === undefined || manifest === undefined || checksum === undefined) return undefined; + return { + archive: archive.digest, + manifest: manifest.digest, + checksum: checksum.digest, + }; +}; + +export const checksumFor = (contents: string, archiveName: string): string | undefined => + contents + .split(/\r?\n/u) + .map((line) => line.trim().match(/^([a-f0-9]{64})\s+[* ]?(.+)$/iu)) + .find((match) => match?.[2] === archiveName || match?.[2]?.endsWith(`/${archiveName}`))?.[1] + ?.toLowerCase(); + +export const manifestMatches = ( + manifestJson: string, + expected: { + readonly service: string; + readonly version: string; + readonly target: string; + }, +): boolean => { + const record = parseJsonRecord(manifestJson); + if (record === undefined) return false; + return ( + record["service"] === expected.service && + record["version"] === expected.version && + record["target"] === expected.target + ); +}; diff --git a/.github/scripts/sync-artifacts-catalog.test.ts b/.github/scripts/sync-artifacts-catalog.test.ts new file mode 100644 index 0000000000..809dce551f --- /dev/null +++ b/.github/scripts/sync-artifacts-catalog.test.ts @@ -0,0 +1,132 @@ +import { describe, expect, test } from "bun:test"; + +import { InvalidPayloadError } from "./slim-mirror-payload.ts"; +import { + CATALOG_PATH, + planArtifactCatalogUpdate, + type ReleasePublication, +} from "./sync-artifacts-catalog.ts"; + +const DIGEST_B = `sha256:${"b".repeat(64)}`; +const POSTGRES_17 = `ghcr.io/supabase/cli/postgres:17.6.1.168@sha256:${"9".repeat(64)}`; +const POSTGRES_15 = `ghcr.io/supabase/cli/postgres:15.14.1.168@sha256:${"f".repeat(64)}`; + +const fixture = `const workloadCatalog = { + database: definition( + "postgres", + "17.6.1.168", + "${POSTGRES_17}", + "bin/supabase-postgres-start", + ["bin/supabase-postgres-start"], + { "15.14.1.168": "${POSTGRES_15}" }, + ), + rest: definition("postgrest", "v16.2", "ghcr.io/supabase/cli/postgrest:v16.2", "bin/postgrest"), + analytics: definition( + "analytics", + "v1.50.9", + "ghcr.io/supabase/cli/analytics:v1.50.9@sha256:${"a".repeat(64)}", + "bin/logflare", + ), +}; +`; + +const published = (digest?: string): ReleasePublication => ({ status: "published", digest }); + +describe("planArtifactCatalogUpdate", () => { + test("pins Dockerfile tags that changed and leaves the other postgres line", async () => { + const plan = await planArtifactCatalogUpdate({ + baseDockerfile: `FROM supabase/postgres:17.6.1.168 AS pg +FROM postgrest/postgrest:v16.2 AS postgrest +FROM supabase/logflare:1.50.9 AS logflare +`, + dockerfile: `FROM supabase/postgres:17.6.1.171 AS pg +FROM library/kong:2.8.1 AS kong +FROM postgrest/postgrest:v16.3 AS postgrest +FROM supabase/logflare:1.50.9 AS logflare +`, + catalog: fixture, + publication: async (service) => published(service === "postgres" ? DIGEST_B : undefined), + }); + + expect(plan.updates.map((update) => `${update.service} ${update.version}`)).toEqual([ + "postgres 17.6.1.171", + "postgrest v16.3", + ]); + expect(plan.source).toContain(`"ghcr.io/supabase/cli/postgres:17.6.1.171@${DIGEST_B}"`); + expect(plan.source).toContain(`"${POSTGRES_15}"`); + expect(plan.source).toContain(`"ghcr.io/supabase/cli/postgrest:v16.3"`); + expect(plan.source).toContain(`"v1.50.9"`); + expect(plan.skipped).toEqual([]); + }); + + test("a missing release blocks the commit and OrioleDB does not", async () => { + const movesPostgres = await planArtifactCatalogUpdate({ + baseDockerfile: "FROM supabase/postgres:17.6.1.168 AS pg\n", + dockerfile: `FROM supabase/postgres:17.6.1.171 AS pg +FROM postgrest/postgrest:v16.3 AS postgrest +`, + catalog: fixture, + publication: async () => ({ status: "missing" }), + }); + expect(movesPostgres.updates).toEqual([]); + expect(movesPostgres.source).toBe(fixture); + expect(movesPostgres.skipped.every((skip) => skip.blocking)).toBe(true); + + const oriole = await planArtifactCatalogUpdate({ + baseDockerfile: + "FROM supabase/postgres:17.6.1.168 AS pg\nFROM postgrest/postgrest:v16.2 AS postgrest\n", + dockerfile: `FROM supabase/postgres:16.0.0.1-orioledb AS pg +FROM postgrest/postgrest:v16.3 AS postgrest +`, + catalog: fixture, + publication: async () => published(), + }); + expect(oriole.updates.map((update) => update.service)).toEqual(["postgrest"]); + expect(oriole.skipped).toEqual([ + { + alias: "pg", + reason: "pg supabase/postgres:16.0.0.1-orioledb has no slim image.", + blocking: false, + }, + ]); + expect(oriole.source).toContain(`"${POSTGRES_17}"`); + }); + + test("refuses a backward pin and a tag that would escape the catalog string", async () => { + const backward = await planArtifactCatalogUpdate({ + baseDockerfile: "FROM supabase/postgres:17.6.1.170 AS pg\n", + dockerfile: "FROM supabase/postgres:17.6.1.171 AS pg\n", + catalog: fixture.replaceAll("17.6.1.168", "17.6.1.173"), + publication: async () => { + throw new Error("publication lookup"); + }, + }); + expect(backward.updates).toEqual([]); + expect(backward.skipped.map((skip) => skip.blocking)).toEqual([true]); + expect(backward.source).toContain(`"17.6.1.173"`); + + await expect( + planArtifactCatalogUpdate({ + dockerfile: 'FROM supabase/gotrue:v1" AS gotrue\n', + catalog: fixture, + publication: async () => published(), + }), + ).rejects.toThrow(InvalidPayloadError); + }); +}); + +describe("against the real catalog", () => { + test("every slim Dockerfile alias is a catalog entry", async () => { + const dockerfile = await Bun.file("apps/cli/src/shared/services/Dockerfile").text(); + const catalog = await Bun.file(CATALOG_PATH).text(); + const plan = await planArtifactCatalogUpdate({ + dockerfile, + catalog, + publication: async () => published(`sha256:${"a".repeat(64)}`), + }); + + expect( + plan.skipped.filter((skip) => skip.blocking && !skip.reason.includes("older than")), + ).toEqual([]); + }); +}); diff --git a/.github/scripts/sync-artifacts-catalog.ts b/.github/scripts/sync-artifacts-catalog.ts new file mode 100644 index 0000000000..838f2c5b75 --- /dev/null +++ b/.github/scripts/sync-artifacts-catalog.ts @@ -0,0 +1,398 @@ +/** + * Pins `packages/stack/src/Artifacts.ts` to the slim workloads named by + * `apps/cli/src/shared/services/Dockerfile`. Native archive URLs are derived + * from service + version. An entry that already carries a digest keeps one: + * the published `ghcr.io/supabase/cli/:` manifest digest. + * + * Run: `bun .github/scripts/sync-artifacts-catalog.ts [base-dockerfile]` + */ + +import { parseDockerfileServiceImages } from "../../apps/cli/src/shared/services/parse-dockerfile-service-images.ts"; +import { isOrioleImage, slimCatalogPin } from "../../apps/cli/src/shared/services/slim-images.ts"; +import { + DIGEST_PATTERN, + InvalidPayloadError, + SOURCE_REGISTRY, + VERSION_PATTERN, + escapeRegExp, +} from "./slim-mirror-payload.ts"; + +export const CATALOG_PATH = "packages/stack/src/Artifacts.ts"; +const DOCKERFILE_PATH = "apps/cli/src/shared/services/Dockerfile"; +const NATIVE_RELEASES = "https://api.github.com/repos/supabase/slim-services/releases/tags"; + +const SLIM_IMAGE_PREFIX = `${SOURCE_REGISTRY}/`; + +/** Leading numeric component, `v` stripped. Only postgres carries more than one line. */ +function releaseLine(version: string): string { + const withoutPrefix = version.replace(/^[vV]/, ""); + const separator = withoutPrefix.indexOf("."); + return separator === -1 ? withoutPrefix : withoutPrefix.slice(0, separator); +} + +/** True when every numeric prefix of `next` is older than `current`. Equal prefixes are not older. */ +function isOlderRelease(next: string, current: string): boolean { + const nextParts = next.replace(/^[vV]/, "").split("."); + const currentParts = current.replace(/^[vV]/, "").split("."); + const count = Math.max(nextParts.length, currentParts.length); + for (let index = 0; index < count; index++) { + const nextValue = leadingInteger(nextParts[index] ?? "0"); + const currentValue = leadingInteger(currentParts[index] ?? "0"); + if (nextValue === undefined || currentValue === undefined) return false; + if (nextValue !== currentValue) return nextValue < currentValue; + } + return false; +} + +function leadingInteger(part: string): number | undefined { + const match = /^(\d+)/.exec(part); + return match === null ? undefined : Number(match[1]); +} + +export interface CatalogPinUpdate { + readonly service: string; + readonly version: string; + readonly previousVersion: string; + readonly target: "default" | "additional"; +} + +export interface SkippedCatalogPin { + readonly alias: string; + readonly reason: string; + /** OrioleDB has no slim image and must not fail the other pins. */ + readonly blocking: boolean; +} + +export interface CatalogPlan { + readonly source: string; + readonly updates: ReadonlyArray; + readonly skipped: ReadonlyArray; +} + +export type ReleasePublication = + | { readonly status: "published"; readonly digest?: string } + | { readonly status: "missing" | "lookup-failed" }; + +/** `definition("", "", ""`. */ +function defaultEntryPattern(service: string): RegExp { + const s = escapeRegExp(service); + return new RegExp( + `(definition\\(\\s*"${s}",\\s*")([^"]+)("\\s*,\\s*")(${escapeRegExp(SLIM_IMAGE_PREFIX)}${s}:[^"]+)(")`, + ); +} + +/** Additional release entries: `"": ""`. */ +function additionalEntryPattern(service: string, version?: string): RegExp { + const s = escapeRegExp(service); + const key = version === undefined ? `[^"]+` : escapeRegExp(version); + return new RegExp( + `"(${key})"(\\s*:\\s*)"(${escapeRegExp(SLIM_IMAGE_PREFIX)}${s}:[^"]+)"`, + version === undefined ? "g" : "", + ); +} + +function imageHasDigest(image: string): boolean { + return /@sha256:[0-9a-f]{64}$/.test(image); +} + +function desiredImage( + service: string, + version: string, + currentImage: string, + digest: string, +): string { + const tagged = `${SLIM_IMAGE_PREFIX}${service}:${version}`; + if (!imageHasDigest(currentImage)) return tagged; + if (!DIGEST_PATTERN.test(digest)) { + throw new InvalidPayloadError(`missing slim digest for ${service}:${version}`); + } + return `${tagged}@${digest}`; +} + +type SelectedEntry = + | { readonly kind: "default" | "additional"; readonly version: string; readonly image: string } + | { readonly kind: "unmodelled-service" } + | { readonly kind: "unmodelled-release-line"; readonly known: ReadonlyArray }; + +function selectEntry(source: string, service: string, version: string): SelectedEntry { + const defaultMatch = defaultEntryPattern(service).exec(source); + if (defaultMatch === null) return { kind: "unmodelled-service" }; + + const currentDefaultVersion = defaultMatch[2] ?? ""; + const currentDefaultImage = defaultMatch[4] ?? ""; + const additional = [...source.matchAll(additionalEntryPattern(service))].map((match) => ({ + version: match[1] ?? "", + image: match[3] ?? "", + })); + const bumpsDefault = + additional.length === 0 || releaseLine(version) === releaseLine(currentDefaultVersion); + if (bumpsDefault) { + return { kind: "default", version: currentDefaultVersion, image: currentDefaultImage }; + } + + const sameLine = additional.find((entry) => releaseLine(entry.version) === releaseLine(version)); + if (sameLine === undefined) { + return { + kind: "unmodelled-release-line", + known: [currentDefaultVersion, ...additional.map((entry) => entry.version)], + }; + } + return { kind: "additional", version: sameLine.version, image: sameLine.image }; +} + +function skipReason(alias: string, version: string, entry: SelectedEntry): string | undefined { + if (entry.kind === "unmodelled-service") { + return `${CATALOG_PATH} has no slim entry for ${alias}.`; + } + if (entry.kind === "unmodelled-release-line") { + return `${alias} ${version} is not on a release line ${CATALOG_PATH} carries (${entry.known.join(", ")}).`; + } + return undefined; +} + +function slimVersions(dockerfile: string): ReadonlyMap { + const versions = new Map(); + for (const from of parseDockerfileServiceImages(dockerfile)) { + const pin = slimCatalogPin(from.alias, from.image); + if (pin !== undefined) versions.set(from.alias, pin.version); + } + return versions; +} + +type PinResult = + | { + readonly kind: "updated"; + readonly source: string; + readonly previousVersion: string; + readonly target: "default" | "additional"; + } + | { readonly kind: "unchanged" }; + +function pinService( + source: string, + service: string, + version: string, + digest: string | undefined, +): PinResult { + const entry = selectEntry(source, service, version); + if (entry.kind !== "default" && entry.kind !== "additional") { + throw new InvalidPayloadError(`${service} ${version} is not a catalog entry.`); + } + if (imageHasDigest(entry.image) && (digest === undefined || !DIGEST_PATTERN.test(digest))) { + throw new InvalidPayloadError(`missing slim digest for ${service}:${version}`); + } + const desired = desiredImage(service, version, entry.image, digest ?? ""); + if (entry.version === version && entry.image === desired) return { kind: "unchanged" }; + + if (entry.kind === "default") { + return { + kind: "updated", + source: source.replace( + defaultEntryPattern(service), + (_full, prefix: string, _version: string, mid: string, _image: string, suffix: string) => + `${prefix}${version}${mid}${desired}${suffix}`, + ), + previousVersion: entry.version, + target: "default", + }; + } + + return { + kind: "updated", + source: source.replace( + additionalEntryPattern(service, entry.version), + (_full, _key: string, separator: string) => `"${version}"${separator}"${desired}"`, + ), + previousVersion: entry.version, + target: "additional", + }; +} + +/** + * Rewrites `catalog` from Dockerfile tags that differ from `baseDockerfile`. + * With no base, every slim tag is in scope. A pin that cannot be applied is + * reported in `skipped`. OrioleDB is the only non-blocking skip. + */ +export async function planArtifactCatalogUpdate(input: { + readonly dockerfile: string; + readonly baseDockerfile?: string; + readonly catalog: string; + readonly publication: (service: string, version: string) => Promise; +}): Promise { + let source = input.catalog; + const updates: CatalogPinUpdate[] = []; + const skipped: SkippedCatalogPin[] = []; + const baseVersions = + input.baseDockerfile === undefined ? undefined : slimVersions(input.baseDockerfile); + const changed = (alias: string, version: string | undefined): boolean => + baseVersions === undefined || baseVersions.get(alias) !== version; + + for (const from of parseDockerfileServiceImages(input.dockerfile)) { + if (isOrioleImage(from.image)) { + if (!changed(from.alias, undefined)) continue; + skipped.push({ + alias: from.alias, + reason: `${from.alias} ${from.image} has no slim image.`, + blocking: false, + }); + continue; + } + const pin = slimCatalogPin(from.alias, from.image); + if (pin === undefined || !changed(from.alias, pin.version)) continue; + if (!VERSION_PATTERN.test(pin.version)) { + throw new InvalidPayloadError(`invalid version for ${from.alias}: '${pin.version}'`); + } + + const entry = selectEntry(source, pin.service, pin.version); + const reason = skipReason(from.alias, pin.version, entry); + if (reason !== undefined) { + skipped.push({ alias: from.alias, reason, blocking: true }); + continue; + } + if (entry.kind !== "default" && entry.kind !== "additional") continue; + if (isOlderRelease(pin.version, entry.version)) { + skipped.push({ + alias: from.alias, + reason: `${pin.service} ${pin.version} is older than the catalog pin ${entry.version}.`, + blocking: true, + }); + continue; + } + if (entry.version === pin.version && !imageHasDigest(entry.image)) continue; + + const release = await input.publication(pin.service, pin.version); + if (release.status !== "published") { + skipped.push({ + alias: from.alias, + reason: + release.status === "missing" + ? `${pin.service}:${pin.version} has no published slim image and native release.` + : `${pin.service}:${pin.version} publication check failed.`, + blocking: true, + }); + continue; + } + const digest = imageHasDigest(entry.image) ? release.digest : undefined; + if (imageHasDigest(entry.image) && (digest === undefined || !DIGEST_PATTERN.test(digest))) { + skipped.push({ + alias: from.alias, + reason: `${pin.service}:${pin.version} has no published slim manifest.`, + blocking: true, + }); + continue; + } + const result = pinService(source, pin.service, pin.version, digest); + if (result.kind === "unchanged") continue; + source = result.source; + updates.push({ + service: pin.service, + version: pin.version, + previousVersion: result.previousVersion, + target: result.target, + }); + } + + return { source, updates, skipped }; +} + +type Probe = "published" | "missing" | "lookup-failed"; + +async function probeManifest(reference: string): Promise<{ probe: Probe; digest?: string }> { + const proc = Bun.spawn(["regctl", "manifest", "head", reference], { + stdout: "pipe", + stderr: "pipe", + }); + const [stdout, stderr, exit] = await Promise.all([ + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + proc.exited, + ]); + const digest = stdout.trim(); + if (exit === 0 && DIGEST_PATTERN.test(digest)) return { probe: "published", digest }; + const detail = stderr.toLowerCase(); + if ( + detail.includes("manifest unknown") || + detail.includes("name unknown") || + detail.includes("not found") || + detail.includes("404") + ) { + return { probe: "missing" }; + } + const message = stderr.trim(); + console.log( + `::warning ::${reference} publication check failed${message === "" ? "" : `: ${message}`}`, + ); + return { probe: "lookup-failed" }; +} + +async function probeNativeRelease(service: string, version: string): Promise { + const tag = `${service}-${version}`; + const headers: Record = { + Accept: "application/vnd.github+json", + "User-Agent": "supabase-cli-catalog-sync", + }; + const token = process.env.GITHUB_TOKEN; + if (token !== undefined && token !== "") headers.Authorization = `Bearer ${token}`; + try { + const response = await fetch(`${NATIVE_RELEASES}/${encodeURIComponent(tag)}`, { headers }); + if (response.status === 200) return "published"; + if (response.status === 404) return "missing"; + console.log(`::warning ::${tag} native release check returned HTTP ${response.status}.`); + return "lookup-failed"; + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + console.log(`::warning ::${tag} native release check failed: ${message}`); + return "lookup-failed"; + } +} + +async function lookupPublication(service: string, version: string): Promise { + const reference = `${SLIM_IMAGE_PREFIX}${service}:${version}`; + const [manifest, native] = await Promise.all([ + probeManifest(reference), + probeNativeRelease(service, version), + ]); + if (manifest.probe === "lookup-failed" || native === "lookup-failed") { + return { status: "lookup-failed" }; + } + if (manifest.probe === "missing" || native === "missing") return { status: "missing" }; + return { status: "published", digest: manifest.digest }; +} + +async function main(argv: ReadonlyArray): Promise { + const [dockerfilePath = DOCKERFILE_PATH, catalogPath = CATALOG_PATH, baseDockerfilePath] = argv; + const dockerfile = await Bun.file(dockerfilePath).text(); + const catalog = await Bun.file(catalogPath).text(); + const baseDockerfile = + baseDockerfilePath === undefined ? undefined : await Bun.file(baseDockerfilePath).text(); + const plan = await planArtifactCatalogUpdate({ + dockerfile, + baseDockerfile, + catalog, + publication: lookupPublication, + }); + for (const skip of plan.skipped) { + console.log(`::warning ::Left ${skip.alias} unchanged: ${skip.reason}`); + } + if (plan.skipped.some((skip) => skip.blocking)) { + console.log("::error ::Refusing to commit a partial catalog update."); + process.exit(1); + } + await Bun.write(catalogPath, plan.source); + if (plan.updates.length === 0) { + console.log("Workload catalog already matches the Dockerfile."); + return; + } + for (const update of plan.updates) { + console.log( + `Pinned ${update.service} ${update.target} ${update.previousVersion} -> ${update.version}.`, + ); + } +} + +if (import.meta.main) { + main(process.argv.slice(2)).catch((error: unknown) => { + console.log(`::error ::${error instanceof Error ? error.message : String(error)}`); + process.exit(1); + }); +} diff --git a/.github/workflows/ai-review.yml b/.github/workflows/ai-review.yml index 879b8fcf1f..5d7792d51a 100644 --- a/.github/workflows/ai-review.yml +++ b/.github/workflows/ai-review.yml @@ -1,17 +1,12 @@ name: AI Review -# One-shot AI code review: replaces the Codex GitHub App's automatic -# per-push reviews (which churned 30-40 short rounds per PR) with a single -# exhaustive pass that runs at most once per PR. See -# .github/ai-review/README.md for the full design and security model. +# One-shot AI code review: replaces the Codex GitHub App's automatic per-push reviews with a +# single exhaustive pass that runs at most once per PR. See .github/ai-review/README.md for the +# full design and security model. # -# Three ways to trigger a run: -# - workflow_dispatch, for testing / ad-hoc runs against any PR number. -# - an internal maintainer commenting `/ai-review` on a PR. -# - automatically, when a PR opens or leaves draft. resolve.ts gates the -# automatic path to PR authors with repository write access; external -# contributors' PRs are skipped and go through the manual `/ai-review` -# maintainer path instead. +# Triggered by workflow_dispatch (testing/ad-hoc), an internal maintainer commenting +# `/ai-review`, or automatically when a PR opens or leaves draft — resolve.ts gates the automatic +# path to PR authors with write access; external contributors go through the manual path instead. on: workflow_dispatch: inputs: @@ -29,30 +24,21 @@ on: permissions: {} -# One source of truth for the two model names — `resolve`/`claude-review`/ -# `codex-review` all read these instead of hardcoding them a second and -# third time, and `post-review`'s footer reads them too (see the "Post -# review" step below). +# One source of truth for the model names; resolve, claude-review, codex-review, and +# post-review's footer all read these instead of hardcoding them separately. env: - CLAUDE_MODEL: claude-opus-5 - CODEX_MODEL: gpt-5.6-sol - -# Ordinary (non-command) issue_comment events fire this workflow for EVERY -# comment on EVERY PR; with only the PR number in the group, any comment -# (even one that isn't `/ai-review`) would cancel an in-flight review via -# `cancel-in-progress`. Give those runs their own per-run group so they can -# never cancel a real review. The command test is exact equality -# (`!= '/ai-review'`), mirroring resolve.ts's first-line check — `startsWith` -# would let a near-miss like `/ai-reviewers` (which resolve.ts rejects) land -# in a shared group and cancel a running review anyway. + CLAUDE_MODEL: claude-opus-5-5 + CODEX_MODEL: gpt-6-sol + +# Non-command issue_comment events fire for every comment on every PR, so grouping by PR number +# alone would let any comment cancel an in-flight review; give those runs their own per-run +# group instead. The command check is exact equality (mirroring resolve.ts), so a near-miss like +# `/ai-reviewers` can't land in the shared group and cancel a real review. # -# `pull_request` events get their own per-PR `auto` group, separate from the -# manual (`/ai-review` / dispatch) `review` group: an auto event may well -# resolve to a SKIP (dedup, no write access), and letting it share the manual -# group would let e.g. a ready_for_review event cancel an in-flight -# maintainer-requested review and then not replace it. The cost is that an -# auto and a manual run can overlap on the same PR — rare, and self-healing, -# since the later post supersedes the earlier review. +# `pull_request` events get a separate per-PR `auto` group from the manual `review` group, since +# an auto event may resolve to a skip and would otherwise cancel an in-flight maintainer-requested +# review without replacing it. An auto and manual run can overlap on the same PR as a result — +# rare, and self-healing since the later post supersedes the earlier one. concurrency: group: >- ai-review-${{ github.event.pull_request.number || github.event.issue.number || inputs.pr }}-${{ @@ -66,21 +52,17 @@ jobs: name: Resolve runs-on: ubuntu-latest timeout-minutes: 5 - # For issue_comment events, only PR comments starting with /ai-review - # AND carrying an association that could plausibly be a maintainer reach - # this job at all. This is a cheap, non-authoritative pre-filter - # (defense-in-depth only): it can't see a private org member's real - # permission, so it can under-admit. The authoritative checks — the - # EXACT command match and the effective-permission lookup — happen in - # resolve.ts, which is the actual gate. + # Only PR comments starting with /ai-review, from an association that could plausibly be a + # maintainer, reach this job — a cheap, non-authoritative pre-filter (it can't see a private + # org member's real permission, so it can under-admit). The authoritative checks — exact + # command match and effective-permission lookup — happen in resolve.ts. if: > github.event_name != 'issue_comment' || (github.event.issue.pull_request != null && startsWith(github.event.comment.body, '/ai-review') && contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association)) - # `resolve` reacts 👀 to the triggering comment (pull-requests: write) but - # runs ONLY trusted, default-branch code (see the pinned checkout ref - # below) — never a PR's own code — so granting it write is safe. + # Reacts 👀 to the triggering comment (pull-requests: write) but runs only trusted, + # default-branch code (see the pinned checkout ref below), never a PR's own — safe to grant write. permissions: pull-requests: write contents: read @@ -90,9 +72,8 @@ jobs: head_ref: ${{ steps.resolve.outputs.head_ref }} trigger: ${{ steps.resolve.outputs.trigger }} steps: - # Base repo, default ref, pinned explicitly — this job runs trusted - # repository code exclusively, and must keep doing so even though the - # `pull_request` trigger above hands it PR-authored event payloads. + # Pinned to the default ref explicitly — this job must keep running only trusted repo code, + # even though the pull_request trigger above hands it PR-authored event payloads. - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: ${{ github.event.repository.default_branch }} @@ -119,14 +100,11 @@ jobs: if: needs.resolve.outputs.should_run == 'true' runs-on: ubuntu-latest timeout-minutes: 60 - # SECURITY-CRITICAL: this job checks out the PR's own head commit, which - # is untrusted review subject matter, not something this job trusts with - # more access. Nothing this job EXECUTES may come from that checkout: - # prompts, the findings schema, and the validation script are all read - # from a SEPARATE trusted checkout of the default branch (`path: trusted` - # below). The job holds no write permissions, a read-only Claude tool - # allowlist (no write/edit tools, no Bash), and no secrets beyond - # ANTHROPIC_API_KEY. + # Security-critical: this job checks out the PR's own head commit as untrusted review subject + # matter only — nothing it executes may come from that checkout. Prompts, the findings schema, + # and the validation script are all read from a separate trusted default-branch checkout + # (`path: trusted` below). The job holds no write permissions, a read-only Claude tool + # allowlist (no write/edit tools, no Bash), and no secrets beyond ANTHROPIC_API_KEY. permissions: contents: read pull-requests: read @@ -169,11 +147,11 @@ jobs: base_ref=$(jq -r '.baseRefName' /tmp/ai-review/pr.json) bun .github/scripts/ai-review/generate-pr-diff.ts "$PR" "$base_ref" - # Pin the exact published version so a new Claude Code release can't - # silently change review behavior mid-rollout; bump deliberately. - # Install from the TRUSTED checkout with npm config isolation so a - # PR-supplied `.npmrc`/`.npmrc`-adjacent config in the untrusted `pr` - # checkout can never redirect this install to a hostile registry. + # Pin the exact published version so a Claude Code release can't silently change review + # behavior mid-rollout. Installed from the trusted checkout with npm config isolation so a + # PR-supplied `.npmrc` in the untrusted `pr` checkout can never redirect this install to a + # hostile registry. + # 2.1.280 is the first release that accepts claude-opus-5-5. - name: Install Claude Code CLI working-directory: trusted run: | @@ -185,20 +163,15 @@ jobs: npm install -g \ --userconfig "${RUNNER_TEMP}/ai-review-npmrc-user" \ --globalconfig "${RUNNER_TEMP}/ai-review-npmrc-global" \ - --registry=https://registry.npmjs.org/ @anthropic-ai/claude-code@2.1.247 - - # SECURITY-CRITICAL invariant: PR code is only ever READ by `claude`, - # via the `( cd .../pr && claude ... )` subshell below — nothing else in - # this step, and no `bun` process anywhere in this job, ever runs with - # a cwd inside `pr`. `bun` auto-loads `bunfig.toml` (`preload` runs - # arbitrary code) and `.env` from its cwd; a `pr`-cwd `bun` invocation - # would let a PR-authored `pr/bunfig.toml` execute attacker code in a - # step that holds `ANTHROPIC_API_KEY`. `claude` is a standalone binary - # (not run via `bun`), so `bunfig.toml` never applies to it; `--bare` - # already disables hooks/MCP/CLAUDE.md, and `--strict-mcp-config` is - # belt-and-suspenders against a future CLI regression. The step's own - # `working-directory: trusted` keeps `jq` and `bun` on the trusted - # checkout for everything outside that one subshell. + --registry=https://registry.npmjs.org/ @anthropic-ai/claude-code@2.1.280 + + # Security-critical invariant: PR code is only ever read by `claude`, in the `pr`-cwd + # subshell below — no other command in this job, including any `bun` call, runs with a cwd + # inside `pr`. `bun` auto-loads `bunfig.toml` (arbitrary `preload` code) and `.env` from its + # cwd, so a `pr`-cwd `bun` call would let a PR-authored `bunfig.toml` execute in a step + # holding `ANTHROPIC_API_KEY`. `claude` is a standalone binary unaffected by `bunfig.toml`; + # `--bare` disables hooks/MCP/CLAUDE.md, and `--strict-mcp-config` guards against a future + # CLI regression. - name: Run Claude review working-directory: trusted env: @@ -249,15 +222,10 @@ jobs: exit 1 fi - # Scrubs any secret-shaped substring a prompt-injected model might have - # echoed back (e.g. from `Read`-ing a secret-bearing path) out of the - # raw JSON before it's uploaded as a (public-repo) artifact; the posted - # review is scrubbed separately at render time. `if: always()` so a - # partial `claude-raw.json` from a failed attempt is still scrubbed - # before the always-on upload step below; guarded because - # `claude-findings.json` may not exist if every attempt failed before - # the extraction step. Runs from the trusted cwd, same as every other - # `bun` invocation in this job. + # Scrubs any secret-shaped substring a prompt-injected model might have echoed back before + # the raw JSON is uploaded as a public-repo artifact; the posted review is scrubbed + # separately at render time. Runs with `if: always()` so a partial `claude-raw.json` from a + # failed attempt is still scrubbed before the always-on upload step. - name: Redact secrets from Claude findings if: always() working-directory: trusted @@ -284,11 +252,10 @@ jobs: name: Codex review needs: resolve if: needs.resolve.outputs.should_run == 'true' - # Codex's INDEPENDENT review. It no longer depends on claude-review, so it - # runs IN PARALLEL with it. It works purely from /tmp/ai-review/pr.diff - # (absolute path in its prompt), so it needs no PR-head checkout — its ONLY - # checkout is the trusted default branch. The verify-by-reading step (which - # does need the PR's files) is the separate `adjudicate` job below. + # Codex's independent review doesn't depend on claude-review, so it runs in parallel. It + # works purely from /tmp/ai-review/pr.diff, so it needs only the trusted default-branch + # checkout — no PR-head checkout. The verify-by-reading step (which needs the PR's files) is + # the separate `adjudicate` job below. permissions: contents: read pull-requests: read @@ -321,18 +288,19 @@ jobs: mkdir -p /tmp/ai-review jq 'del(.["$schema"])' .github/ai-review/findings.schema.json > /tmp/ai-review/findings.schema.json - # Safety strategy (drop-sudo + read-only), verified against the pinned - # openai/codex-action@52fe01ec…'s action.yml + src/runCodexExec.ts — see - # the adjudicate job below for the full rationale. In short: Codex runs as - # a non-sudo-capable user, in a sandbox with no filesystem writes and no - # network, with no `codex-args`/`--sandbox` duplication. + # Safety strategy (drop-sudo + read-only) — see the "Run Codex adjudication" step below for + # the full rationale (verified against the pinned openai/codex-action's action.yml + + # src/runCodexExec.ts). In short: non-sudo user, no filesystem writes, no network. - name: Run Codex independent review - # Pinned to v1.11, NOT v1.12: v1.12 has a confirmed regression where a - # heavy Linux run never returns after Codex finishes the turn and writes - # its output file — the step sits idle until the job timeout, discarding - # a completed review (openai/codex-action#150). v1.11 handles the same - # heavy workload cleanly. There is no released fix above v1.12 yet. - uses: openai/codex-action@86365089eb2b84e0a8fb0717b304f8bdcb13b20e # v1.12 + # Pinned to v1.11 (52fe01ec…), not v1.12 (86365089…): v1.12 has two open upstream + # regressions on this workflow's config (safety-strategy: drop-sudo, sandbox: read-only, + # output-schema-file) — openai/codex-action#151 (a lingering descendant process keeps the + # step alive after Codex already wrote its output) and openai/codex-action#160 + # (drop-sudo's rewrite chmods root-owned /run sockets, breaking systemd-resolved and + # killing the job regardless of timeout-minutes). A dependabot.yml ignore entry for + # openai/codex-action stops an automated bump from silently regressing this pin; + # re-verify both issues are closed before ever re-bumping it. + uses: &codex-action-pin openai/codex-action@52fe01ec70a42f454c9d2ebd47598f9fd6893d56 # v1.11 with: openai-api-key: ${{ secrets.OPENAI_API_KEY }} prompt-file: .github/ai-review/codex-review-prompt.md @@ -340,9 +308,8 @@ jobs: effort: high output-schema-file: /tmp/ai-review/findings.schema.json output-file: /tmp/ai-review/codex-findings.json - # Pinned explicitly (verified via `npm view @openai/codex version`); - # never left floating. - codex-version: "0.150.1" + # 0.156.1 is the first stable CLI whose model catalog carries gpt-6-sol. + codex-version: &codex-cli-version "0.156.1" working-directory: ${{ github.workspace }} safety-strategy: drop-sudo sandbox: read-only @@ -377,21 +344,17 @@ jobs: - resolve - claude-review - codex-review - # Runs when AT LEAST ONE independent review succeeded — a single flaky model - # job must not sink the whole review. Each findings download below is guarded - # by its job's result, and the stage step substitutes an empty findings set - # for any review that didn't complete, so the adjudicator reconciles 1 or 2. + # Runs when at least one independent review succeeded — a single flaky model job must not + # sink the whole review. Findings download steps are guarded per job result, and staging + # substitutes an empty findings set for any review that didn't complete. if: ${{ !cancelled() && needs.resolve.outputs.should_run == 'true' && (needs.claude-review.result == 'success' || needs.codex-review.result == 'success') }} - # SECURITY-CRITICAL: this job checks out the PR head (untrusted subject - # matter) so Codex can VERIFY findings by reading the real files. Codex runs - # with its working directory at the workspace ROOT, which holds only the - # `pr/` and `trusted/` checkouts (no AGENTS.md/config of its own), and reads - # `pr/` read-only; the adjudicate prompt's injection guard treats every file - # under `pr/` (including any AGENTS.md/CLAUDE.md) as untrusted data. Every - # `bun` invocation runs from `trusted/`. Blast radius of a prompt-injected - # Codex here is bounded to review CONTENT: read-only sandbox, no network, - # key proxied by the action, and the output is secret-scrubbed before it - # leaves this job. + # Security-critical: this job checks out the PR head (untrusted subject matter) so Codex can + # verify findings by reading the real files. Codex's working directory is the workspace root, + # which holds only `pr/` and `trusted/` (no AGENTS.md/config of its own); it reads `pr/` + # read-only, and the adjudicate prompt's injection guard treats every file under `pr/` + # (including any AGENTS.md/CLAUDE.md) as untrusted data. Every `bun` invocation runs from + # `trusted/`. A prompt-injected Codex here is bounded to review content: read-only sandbox, + # no network, key proxied by the action, and secret-scrubbed output. permissions: contents: read pull-requests: read @@ -473,33 +436,23 @@ jobs: mkdir -p /tmp/ai-review jq 'del(.["$schema"])' .github/ai-review/merged-review.schema.json > /tmp/ai-review/merged-review.schema.json - # Safety strategy, verified against the pinned - # openai/codex-action@52fe01ec…'s action.yml + src/runCodexExec.ts: - # - `safety-strategy: read-only` forces codex-exec's legacy sandbox to - # read-only, but Codex still runs as the action's default, - # sudo-capable user — the action's own docs/security.md calls this - # combination out as unsafe, since a sudo-capable process can read - # secrets like OPENAI_API_KEY out of memory (e.g. via procfs) even - # under a read-only filesystem sandbox with no network. - # - `safety-strategy: drop-sudo` (the action's default) removes sudo - # from the user running Codex, closing that hole, but says nothing - # on its own about Codex's filesystem/network sandbox. - # - `determinePermissionSelection()` only forces the legacy read-only - # sandbox when `safety-strategy === "read-only"`; otherwise it honors - # a separately-set `sandbox` input as-is. So setting BOTH - # `safety-strategy: drop-sudo` and `sandbox: read-only` composes them - # safely: non-sudo user, no filesystem writes, no network — with no - # `codex-args`/`--sandbox` duplication. - # `working-directory` is the workspace root so Codex's cwd holds no - # untrusted AGENTS.md/config; it reads the PR from `pr/` and executes - # nothing from it. + # Safety strategy, per the pinned openai/codex-action's action.yml + src/runCodexExec.ts: + # - `safety-strategy: read-only` forces the legacy sandbox read-only, but Codex still + # runs as the action's default sudo-capable user — the action's own docs call this + # unsafe, since a sudo-capable process can read secrets like OPENAI_API_KEY out of + # memory even under a read-only, no-network sandbox. + # - `safety-strategy: drop-sudo` (the action's default) removes sudo from the user + # running Codex but says nothing on its own about the filesystem/network sandbox. + # - The action only forces its legacy read-only sandbox when `safety-strategy === + # "read-only"`; otherwise it honors a separately-set `sandbox` input as-is. Setting + # `safety-strategy: drop-sudo` and `sandbox: read-only` together composes them safely: + # non-sudo user, no filesystem writes, no network. + # `working-directory` is the workspace root so Codex's cwd holds no untrusted AGENTS.md/ + # config; it reads the PR from `pr/` and executes nothing from it. - name: Run Codex adjudication - # Pinned to v1.11, NOT v1.12: v1.12 has a confirmed regression where a - # heavy Linux run never returns after Codex finishes the turn and writes - # its output file — the step sits idle until the job timeout, discarding - # a completed review (openai/codex-action#150). v1.11 handles the same - # heavy workload cleanly. There is no released fix above v1.12 yet. - uses: openai/codex-action@86365089eb2b84e0a8fb0717b304f8bdcb13b20e # v1.12 + # Same v1.11 action pin and Codex CLI version as codex-review's step above (see its + # comments for the rationale); both are YAML-aliased so each changes in one place only. + uses: *codex-action-pin with: openai-api-key: ${{ secrets.OPENAI_API_KEY }} prompt-file: trusted/.github/ai-review/adjudicate-prompt.md @@ -507,7 +460,7 @@ jobs: effort: high output-schema-file: /tmp/ai-review/merged-review.schema.json output-file: /tmp/ai-review/merged-review.json - codex-version: "0.150.1" + codex-version: *codex-cli-version working-directory: ${{ github.workspace }} safety-strategy: drop-sudo sandbox: read-only @@ -540,23 +493,22 @@ jobs: needs: - resolve - adjudicate - # Runs only when adjudication succeeded (it produced the merged review this - # job posts). `!cancelled()` is required here because an explicit `if` - # replaces the default "all needed jobs succeeded" check. + # Runs only when adjudication succeeded (it produced the merged review this job posts). + # `!cancelled()` is required since an explicit `if` replaces the default "all needed jobs + # succeeded" check. if: ${{ !cancelled() && needs.resolve.outputs.should_run == 'true' && needs.adjudicate.result == 'success' }} runs-on: ubuntu-latest timeout-minutes: 10 permissions: pull-requests: write steps: - # SECURITY-CRITICAL: this is the only job with write permission, so it - # must only ever execute trusted base-branch code — never the PR head. - # Checking out `develop` explicitly (never `needs.resolve.outputs.head_ref`) - # keeps a malicious PR from smuggling a script change into the one job - # that can write back to the PR. (For `pull_request` events GitHub runs - # the workflow FILE from the PR's own ref; acceptable because the auto - # path only admits same-repo PRs, whose authors hold write access - # anyway, and fork PRs run with a read-only token and no secrets.) + # Security-critical: this is the only job with write permission, so it must only ever + # execute trusted base-branch code — never the PR head. Checking out `develop` explicitly + # (never `needs.resolve.outputs.head_ref`) keeps a malicious PR from smuggling a script + # change into the one job that can write back to the PR. (For `pull_request` events GitHub + # runs the workflow file from the PR's own ref; acceptable since the auto path only admits + # same-repo PRs, whose authors hold write access anyway, and fork PRs run with a read-only + # token and no secrets.) - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: ref: develop @@ -579,7 +531,6 @@ jobs: MERGED_REVIEW_PATH: /tmp/ai-review/merged-review.json TRIGGER: ${{ needs.resolve.outputs.trigger }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - # CLAUDE_MODEL / CODEX_MODEL are inherited from the workflow-level - # `env:` block above — the same values passed to `claude`/ - # `codex-action` — so the footer never drifts from what actually ran. + # CLAUDE_MODEL / CODEX_MODEL are inherited from the workflow-level `env:` block above, + # so the footer never drifts from what actually ran. run: bun .github/scripts/ai-review/post-review.ts post diff --git a/.github/workflows/apply-release-notes.yml b/.github/workflows/apply-release-notes.yml index 054fc20d26..0a0b7e60e0 100644 --- a/.github/workflows/apply-release-notes.yml +++ b/.github/workflows/apply-release-notes.yml @@ -1,14 +1,9 @@ name: Apply release notes -# Approval-based publish. When a member of the supabase/cli team approves a -# release-notes PR (head ref `release-notes/v`), this workflow pushes -# the proposed notes to the GitHub Release body for the corresponding tag, -# comments the release URL on the PR, and closes the PR without merging. The -# release-notes PR targets `develop` (not `main`) so an accidental merge can -# never rewrite `main`'s history; the file is not meant to land on any branch. -# -# Mirrors the fast-forward job in release.yml, which already gates on a -# `pull_request_review` + `approved` event. +# Approval-based publish: when a supabase/cli team member approves a release-notes PR (head ref +# `release-notes/v`), this pushes the notes to the GitHub Release body for that tag, +# comments the release URL on the PR, and closes it without merging. The PR targets `develop` +# (not `main`) so an accidental merge can never rewrite `main`'s history. on: pull_request_review: @@ -32,9 +27,8 @@ jobs: outputs: authorized: ${{ steps.check.outputs.authorized }} steps: - # App token: needs `orgs/.../teams/.../memberships` read (the org-installed - # App has it), repo write to edit the release, and PR write to comment - # and close. Matches release.yml's fast-forward step. + # App token needs org membership read, repo write to edit the release, and PR write to + # comment and close. - id: app-token uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 with: @@ -47,9 +41,8 @@ jobs: GH_TOKEN: ${{ steps.app-token.outputs.token }} APPROVER: ${{ github.event.review.user.login }} PR_NUMBER: ${{ github.event.pull_request.number }} - # Fail closed: any response other than an active membership means the - # approval is ignored. We post a comment so the reviewer sees why their - # approval didn't apply, then exit 0 so the workflow isn't flagged red. + # Fail closed: any state other than active membership is ignored. Exit 0 (after + # commenting why) so the workflow isn't flagged red. run: | set -euo pipefail status=$(gh api \ @@ -79,9 +72,8 @@ jobs: client-id: ${{ vars.GH_APP_CLIENT_ID }} private-key: ${{ secrets.GH_APP_PRIVATE_KEY }} - # Checkout the PR head so any reviewer edits made in the GitHub UI before - # approval are captured. apply-release-notes.ts reads from the working - # tree. + # Checkout the PR head so reviewer edits made before approval are captured; + # apply-release-notes.ts reads from the working tree. - uses: useblacksmith/checkout@6fd481652155169ed4d2f25ebaf97464f685175f # v1 with: ref: ${{ github.event.pull_request.head.sha }} @@ -98,8 +90,7 @@ jobs: HEAD_REF: ${{ github.event.pull_request.head.ref }} PR_NUMBER: ${{ github.event.pull_request.number }} APPROVER: ${{ github.event.review.user.login }} - # The branch is named `release-notes/v`, so the tag is just - # the basename. apply-release-notes.ts validates the file's existence. + # The branch is named `release-notes/v`, so the tag is just the basename. run: | set -euo pipefail tag="${HEAD_REF##release-notes/}" diff --git a/.github/workflows/build-cli-artifacts.yml b/.github/workflows/build-cli-artifacts.yml index d4024a5ae8..2e8c2a32db 100644 --- a/.github/workflows/build-cli-artifacts.yml +++ b/.github/workflows/build-cli-artifacts.yml @@ -7,10 +7,6 @@ on: description: CLI package version to build required: true type: string - shell: - description: CLI shell to package as the shipped supabase binary - required: true - type: string ref: description: Optional git ref or SHA to check out before building required: false @@ -55,7 +51,6 @@ jobs: runs-on: ${{ inputs.runner }} timeout-minutes: ${{ inputs.timeout_minutes }} env: - BUN_SHELL: ${{ inputs.shell }} VERSION: ${{ inputs.version }} SENTRY_DSN: ${{ secrets.SENTRY_DSN }} POSTHOG_API_KEY: ${{ secrets.POSTHOG_API_KEY }} @@ -111,9 +106,9 @@ jobs: - name: Sync versions run: pnpm exec bun apps/cli/scripts/sync-versions.ts --version "${VERSION}" - - name: Build selected shell + - name: Build CLI artifacts timeout-minutes: ${{ inputs.build_timeout_minutes }} - run: pnpm exec bun apps/cli/scripts/build.ts --version "${VERSION}" --shell "${BUN_SHELL}" + run: pnpm exec bun apps/cli/scripts/build.ts --version "${VERSION}" - name: Verify build artifacts run: | @@ -131,7 +126,7 @@ jobs: path: | packages/cli-*/bin/ dist/ - key: cli-build-${{ github.run_id }}-${{ inputs.shell }}-${{ inputs.version }}${{ inputs.cache_key_suffix }}-v1 + key: cli-build-${{ github.run_id }}-${{ inputs.version }}${{ inputs.cache_key_suffix }}-v1 enableCrossOsArchive: true lookup-only: true @@ -142,5 +137,5 @@ jobs: path: | packages/cli-*/bin/ dist/ - key: cli-build-${{ github.run_id }}-${{ inputs.shell }}-${{ inputs.version }}${{ inputs.cache_key_suffix }}-v1 + key: cli-build-${{ github.run_id }}-${{ inputs.version }}${{ inputs.cache_key_suffix }}-v1 enableCrossOsArchive: true diff --git a/.github/workflows/cli-artifacts-infra.yml b/.github/workflows/cli-artifacts-infra.yml new file mode 100644 index 0000000000..28c0cb564e --- /dev/null +++ b/.github/workflows/cli-artifacts-infra.yml @@ -0,0 +1,117 @@ +name: CLI Artifacts Infra + +# Deploys infra/cli-artifacts/template.yaml into the cli-artifacts-prod AWS account: the +# public-read S3 bucket that mirrors slim native artifacts and the PutObject-only role the +# mirror-slim-image workflow assumes. Only develop can assume github-deploy in that account, so +# pull requests lint the template and never touch AWS. Details: infra/cli-artifacts/README.md. + +on: + pull_request: + paths: + - infra/cli-artifacts/** + - .github/workflows/cli-artifacts-infra.yml + push: + branches: + - develop + paths: + - infra/cli-artifacts/** + - .github/workflows/cli-artifacts-infra.yml + workflow_dispatch: + +permissions: {} + +# Lint runs are per pull request or ref; deploys serialize on their own group below. +concurrency: + group: cli-artifacts-infra-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} + +jobs: + lint: + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Install cfn-lint + run: | + set -euo pipefail + python3 -m pip install --user --quiet "cfn-lint==1.57.0" + echo "${HOME}/.local/bin" >> "$GITHUB_PATH" + + - name: Lint template + run: cfn-lint infra/cli-artifacts/template.yaml + + deploy: + needs: lint + if: github.event_name != 'pull_request' && github.ref == 'refs/heads/develop' && vars.CLI_ARTIFACTS_AWS_ACCOUNT_ID != '' + runs-on: ubuntu-latest + timeout-minutes: 20 + concurrency: + group: cli-artifacts-infra-deploy + cancel-in-progress: false + permissions: + contents: read + id-token: write + env: + # Passed explicitly on every deploy: `cloudformation deploy` keeps an existing stack's + # previous parameter values unless they are overridden, so template defaults alone + # would never update a live stack. + BUCKET_NAME: supabase-cli-artifacts + TRUSTED_SUBJECT: repo:supabase/cli:ref:refs/heads/develop + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Configure aws credentials + uses: aws-actions/configure-aws-credentials@cbe3b392738ccf3f987d68400dafcf4b0624a56c # v6.2.4 + with: + role-to-assume: arn:aws:iam::${{ vars.CLI_ARTIFACTS_AWS_ACCOUNT_ID }}:role/github-deploy + aws-region: us-east-1 + + - name: Clear a failed initial create + # A create that rolled back leaves the stack in ROLLBACK_COMPLETE, which cannot be + # updated. Its resources are already gone (the bucket uses RetainExceptOnCreate), so + # deleting the stack record is the only way to retry. + run: | + set -euo pipefail + status="$(aws cloudformation describe-stacks --stack-name cli-artifacts \ + --query "Stacks[0].StackStatus" --output text 2>/dev/null || echo NONE)" + echo "current stack status: ${status}" + if [ "${status}" = "ROLLBACK_COMPLETE" ]; then + aws cloudformation delete-stack --stack-name cli-artifacts + aws cloudformation wait stack-delete-complete --stack-name cli-artifacts + fi + + - name: Deploy stack + run: | + set -euo pipefail + aws cloudformation deploy \ + --stack-name cli-artifacts \ + --template-file infra/cli-artifacts/template.yaml \ + --capabilities CAPABILITY_NAMED_IAM \ + --no-fail-on-empty-changeset \ + --parameter-overrides \ + "BucketName=${BUCKET_NAME}" \ + "TrustedSubject=${TRUSTED_SUBJECT}" + + - name: Show stack events on failure + if: failure() + run: | + set -euo pipefail + aws cloudformation describe-stack-events \ + --stack-name cli-artifacts \ + --query "StackEvents[?contains(ResourceStatus, 'FAILED')].[LogicalResourceId,ResourceStatus,ResourceStatusReason]" \ + --output table || true + + - name: Show stack outputs + run: | + set -euo pipefail + aws cloudformation describe-stacks \ + --stack-name cli-artifacts \ + --query "Stacks[0].Outputs" \ + --output table diff --git a/.github/workflows/cli-go-api-sync.yml b/.github/workflows/cli-go-api-sync.yml index 7efdb8a4c4..7593731443 100644 --- a/.github/workflows/cli-go-api-sync.yml +++ b/.github/workflows/cli-go-api-sync.yml @@ -4,6 +4,12 @@ on: repository_dispatch: types: - api-sync + # Fallback for a lost dispatch, per docs/openapi-sync.md. This matters more + # now that the per-pull-request Codegen check reads the committed snapshot: + # this workflow is the only thing that looks at the live specification, so a + # dropped dispatch would otherwise leave drift undetected indefinitely. + schedule: + - cron: "17 6 * * *" workflow_dispatch: # allow manual triggering permissions: @@ -23,13 +29,30 @@ jobs: go-version-file: apps/cli-go/go.mod cache: true + # The only place that reads the live spec. Every other consumer, including + # the per-PR Codegen check, regenerates from the committed snapshot, so + # upstream drift turns into this one pull request instead of a failure on + # every open pull request that touches apps/cli-go. + - name: Refresh the OpenAPI snapshot + run: curl -fsSL "$API_SPEC_URL" -o api/v1-openapi.yaml + env: + API_SPEC_URL: https://api.supabase.green/api/v1-yaml + - name: Run codegen run: go generate + # Covers the snapshot as well as the generated client: an upstream edit + # that codegen ignores (a description, an example) would otherwise be + # refetched and discarded on every run, leaving the snapshot stale. The + # snapshot is compared byte-for-byte for the same reason -- under + # --ignore-space-at-eol a trailing-whitespace-only upstream edit would be + # refetched and discarded forever, which is the loop this check prevents. + # pkg keeps the flag: it is oapi-codegen output, not upstream bytes. - name: Check for changes id: check run: | - if git diff --ignore-space-at-eol --exit-code --quiet pkg; then + if git diff --exit-code --quiet api/v1-openapi.yaml && + git diff --ignore-space-at-eol --exit-code --quiet pkg; then echo "No changes detected" echo "has_changes=false" >> $GITHUB_OUTPUT else @@ -57,7 +80,7 @@ jobs: body: | This PR was automatically created to sync API types from the infrastructure repository. - Changes were detected in the generated API code after syncing with the latest spec from infrastructure. + Changes were detected after refreshing `apps/cli-go/api/v1-openapi.yaml` from the latest spec from infrastructure, and `apps/cli-go/pkg/api` is regenerated from it. The snapshot diff is the upstream API change and is worth reading: it is marked `linguist-generated`, so GitHub collapses it behind a "Load diff" click rather than hiding it. branch: sync/api-types base: develop diff --git a/.github/workflows/cli-go-ci.yml b/.github/workflows/cli-go-ci.yml index ffa727aea5..a4e60bfab4 100644 --- a/.github/workflows/cli-go-ci.yml +++ b/.github/workflows/cli-go-ci.yml @@ -119,6 +119,12 @@ jobs: go-version-file: apps/cli-go/go.mod cache: true + # Regenerates from the committed api/v1-openapi.yaml snapshot, so this + # check is hermetic: it fails only when a pull request leaves pkg/api out + # of sync with the snapshot it was generated from. It deliberately does + # not compare against the live upstream spec -- doing so made unrelated + # pull requests fail together whenever staging drifted ahead of the + # committed snapshot. That drift is the API Sync workflow's job. - run: go generate - run: | if ! git diff --ignore-space-at-eol --exit-code --quiet pkg; then diff --git a/.github/workflows/cli-go-codeql.yml b/.github/workflows/cli-go-codeql.yml index e079be038c..3f19507312 100644 --- a/.github/workflows/cli-go-codeql.yml +++ b/.github/workflows/cli-go-codeql.yml @@ -1,14 +1,3 @@ -# For most projects, this workflow file will not need changing; you simply need -# to commit it to your repository. -# -# You may wish to alter this file to override the set of languages analyzed, -# or to provide custom queries or build logic. -# -# ******** NOTE ******** -# We have attempted to detect the languages in your repository. Please check -# the `language` matrix defined below to confirm you have the correct set of -# supported CodeQL languages. -# name: "CodeQL" on: @@ -25,11 +14,7 @@ on: jobs: analyze: name: Analyze (${{ matrix.language }}) - # Runner size impacts CodeQL analysis time. To learn more, please see: - # - https://gh.io/recommended-hardware-resources-for-running-codeql - # - https://gh.io/supported-runners-and-hardware-resources - # - https://gh.io/using-larger-runners (GitHub.com only) - # Consider using larger runners or machines with greater resources for possible analysis time improvements. + # Swift requires a macOS runner; every other language in the matrix uses blacksmith. runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'blacksmith-8vcpu-ubuntu-2404' }} timeout-minutes: ${{ (matrix.language == 'swift' && 120) || 360 }} permissions: @@ -51,39 +36,19 @@ jobs: build-mode: autobuild - language: javascript-typescript build-mode: none - # CodeQL supports the following values keywords for 'language': 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift' - # Use `c-cpp` to analyze code written in C, C++ or both - # Use 'java-kotlin' to analyze code written in Java, Kotlin or both - # Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both - # To learn more about changing the languages that are analyzed or customizing the build mode for your analysis, - # see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning. - # If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how - # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - # Initializes the CodeQL tools for scanning. - name: Initialize CodeQL uses: github/codeql-action/init@cdf488f595d80d6e07e03d4674febd5ab45fa938 # v4.37.9 with: languages: ${{ matrix.language }} build-mode: ${{ matrix.build-mode }} - # If you wish to specify custom queries, you can do so here or in a config file. - # By default, queries listed here will override any specified in a config file. - # Prefix the list here with "+" to use these queries and those in the config file. - # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs - # queries: security-extended,security-and-quality - - # If the analyze step fails for one of the languages you are analyzing with - # "We were unable to automatically build your code", modify the matrix above - # to set the build mode to "manual" for that language. Then modify this step - # to build your code. - # ℹ️ Command-line programs to run using the OS shell. - # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun + # Placeholder manual-build step; only runs if a language's build-mode above is set to "manual". - if: matrix.build-mode == 'manual' shell: bash run: | diff --git a/.github/workflows/dispatch-cli-e2e-ci.yml b/.github/workflows/dispatch-cli-e2e-ci.yml index 109b85cf49..ed5aa0f22a 100644 --- a/.github/workflows/dispatch-cli-e2e-ci.yml +++ b/.github/workflows/dispatch-cli-e2e-ci.yml @@ -1,20 +1,9 @@ name: Dispatch cli-e2e-ci -# Asks the supabase/cli-e2e-ci harness to run the cli `test:live` suite against -# a full supabox stack, built from THIS PR's head commit (CLI-1825 / CLI-1831). -# -# This is distinct from `live-e2e.yml`, which runs the collocated live suite -# against managed staging (api.supabase.green). Here the same `apps/cli` suite -# runs against a local Supabox stack stood up inside the private cli-e2e-ci repo; -# we only fire the trigger and pass our head SHA — cli-e2e-ci checks that SHA out -# into its `cli` submodule. -# -# Opt-in by label to keep the expensive full-stack run off every PR: add the -# `run-live-e2e-ci` label (re-dispatches on each subsequent push while labeled). -# cli-e2e-ci reports a `cli-e2e-ci / live` commit status back onto the head SHA. -# -# Fork PRs cannot dispatch (no access to the App secret); run cli-e2e-ci's own -# workflow_dispatch with `cli_ref` for those. +# Triggers supabase/cli-e2e-ci to run the cli `test:live` suite against a full Supabox stack +# built from this PR's head commit — distinct from live-e2e.yml, which targets managed staging. +# Opt in with the `run-live-e2e-ci` label; fork PRs cannot dispatch (no access to the App +# secret) and use cli-e2e-ci's own workflow_dispatch with `cli_ref` instead. on: pull_request: types: [labeled, synchronize, reopened] @@ -24,9 +13,7 @@ permissions: jobs: dispatch: - # Same-repo PRs only: fork PRs don't receive secrets (GH_APP_PRIVATE_KEY), so - # the App-token step would fail and leave a red check. Skip them cleanly — - # fork PRs use cli-e2e-ci's workflow_dispatch with `cli_ref` instead. + # Fork PRs don't receive secrets (GH_APP_PRIVATE_KEY), so the App-token step would fail; skip them. if: >- contains(github.event.pull_request.labels.*.name, 'run-live-e2e-ci') && github.event.pull_request.head.repo.full_name == github.repository diff --git a/.github/workflows/github-scripts-ci.yml b/.github/workflows/github-scripts-ci.yml index ada4560ee1..c2f12fa74b 100644 --- a/.github/workflows/github-scripts-ci.yml +++ b/.github/workflows/github-scripts-ci.yml @@ -9,8 +9,13 @@ on: pull_request: paths: - ".github/scripts/**" + - "apps/cli/src/shared/services/Dockerfile" + - "apps/cli/src/shared/services/parse-dockerfile-service-images.ts" + - "apps/cli/src/shared/services/slim-images.ts" + - "packages/stack/src/Artifacts.ts" - ".github/workflows/ai-review.yml" - ".github/workflows/github-scripts-ci.yml" + - ".github/workflows/mirror-slim-image.yml" permissions: {} @@ -46,11 +51,12 @@ jobs: - name: Run tests run: | set -uo pipefail - # The leading "./" is load-bearing: `bun test .github/scripts` - # (without it) silently discovers ZERO tests and still exits 0. - # Capture output to a file instead of piping it, so `test_exit` - # below is `bun test`'s own exit code, not `tee`/`grep`'s. - bun test ./.github/scripts > /tmp/github-scripts-test-output.txt 2>&1 + # The root `test:github-scripts` script runs `bun test ./.github/scripts`; + # its leading "./" is load-bearing, since `bun test .github/scripts` + # silently discovers ZERO tests and still exits 0. Capture output to a + # file instead of piping it, so `test_exit` below is the script's own + # exit code, not `tee`/`grep`'s. + pnpm run test:github-scripts > /tmp/github-scripts-test-output.txt 2>&1 test_exit=$? cat /tmp/github-scripts-test-output.txt if [ "$test_exit" -ne 0 ]; then diff --git a/.github/workflows/lint-pull-request.yml b/.github/workflows/lint-pull-request.yml index c70377a830..86106fad52 100644 --- a/.github/workflows/lint-pull-request.yml +++ b/.github/workflows/lint-pull-request.yml @@ -3,9 +3,9 @@ name: Lint Pull Request # Release-notes PRs (head ref `release-notes/*`) skip CI; only # apply-release-notes.yml runs for those. # -# Draft PRs skip this check unless they carry `run-ci` (same opt-in as Test -# and preview packages). Label events retrigger this cheap check so adding -# `run-ci` on a draft starts lint without waiting for a push. +# Draft PRs skip this check unless they carry `run-ci` (same opt-in as Test). +# Label events retrigger this cheap check so adding `run-ci` on a draft starts +# lint without waiting for a push. on: pull_request_target: types: @@ -44,5 +44,22 @@ jobs: uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Keep this scope list in sync with commitlint.config.js. + with: + scopes: | + api + cli + cli-e2e + cli-go + cli-test-helpers + config + docs + process-compose + stack + ci + deps + repo + misc + release - if: github.event_name == 'merge_group' run: echo "Merge queue entry does not have a pull request payload; reporting success for the required lint check." diff --git a/.github/workflows/live-e2e-gate.yml b/.github/workflows/live-e2e-gate.yml new file mode 100644 index 0000000000..80c5f8985b --- /dev/null +++ b/.github/workflows/live-e2e-gate.yml @@ -0,0 +1,85 @@ +name: Live E2E Gate + +on: + workflow_call: + inputs: + sha: + description: Exact commit to verify + required: true + type: string + secrets: + SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN: + required: true + +permissions: + actions: read + contents: read + +jobs: + lookup: + name: Find exact-SHA live result + runs-on: ubuntu-latest + timeout-minutes: 5 + outputs: + reuse: ${{ steps.lookup.outputs.reuse }} + steps: + - name: Check latest relevant run + id: lookup + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + SHA: ${{ inputs.sha }} + run: | + set -euo pipefail + response="$(gh api --paginate "/repos/${REPOSITORY}/actions/workflows/live-e2e.yml/runs?branch=develop&head_sha=${SHA}&per_page=100")" + latest="$(jq -s -c 'map(.workflow_runs[]) | sort_by(.run_number) | reverse | .[0] // empty' <<< "$response")" + if [[ -z "$latest" ]]; then + echo "No live run exists for ${SHA}; the stable gate will run the suite." + echo "reuse=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + status="$(jq -r '.status' <<< "$latest")" + conclusion="$(jq -r '.conclusion // empty' <<< "$latest")" + run_id="$(jq -r '.id' <<< "$latest")" + echo "Latest live run ${run_id}: status=${status}, conclusion=${conclusion}" + jobs="$(gh api --paginate "/repos/${REPOSITORY}/actions/runs/${run_id}/jobs?per_page=100")" + # Keep this exact job suffix and step name aligned with the producer + # and notifier; renaming either changes reuse semantics. + executed="$(jq -s -r 'any(.[].jobs[]; (.name == "Live e2e" or (.name | endswith(" / Live e2e"))) and .conclusion == "success" and any(.steps[]?; .name == "Run live e2e" and .conclusion == "success"))' <<< "$jobs")" + if [[ "$status" == completed && "$conclusion" == success && "$executed" == true ]]; then + echo "reuse=true" >> "$GITHUB_OUTPUT" + else + echo "Latest live run did not pass; the stable gate will run the suite." + echo "reuse=false" >> "$GITHUB_OUTPUT" + fi + + run: + name: Run live e2e for stable gate + needs: lookup + if: needs.lookup.outputs.reuse != 'true' + uses: ./.github/workflows/live-e2e-suite.yml + with: + ref: ${{ inputs.sha }} + concurrency_group: live-e2e-stable-${{ inputs.sha }} + # Forward the one secret required by live-e2e-suite.yml. + secrets: + SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} + + result: + name: Verify live e2e gate + needs: [lookup, run] + if: ${{ !cancelled() }} + runs-on: ubuntu-latest + timeout-minutes: 2 + steps: + - name: Require a successful result + env: + REUSE: ${{ needs.lookup.outputs.reuse }} + RUN_RESULT: ${{ needs.run.result }} + run: | + set -euo pipefail + if [[ "$REUSE" == true || "$RUN_RESULT" == success ]]; then + exit 0 + fi + echo "::error::live e2e did not produce a successful result" + exit 1 diff --git a/.github/workflows/live-e2e-notify.yml b/.github/workflows/live-e2e-notify.yml new file mode 100644 index 0000000000..5307ebb2ec --- /dev/null +++ b/.github/workflows/live-e2e-notify.yml @@ -0,0 +1,123 @@ +name: Live E2E Notifications + +on: + workflow_run: + workflows: [Live E2E, Release] + types: [completed] + +permissions: + actions: read + contents: read + +jobs: + notify: + if: github.event.workflow_run.conclusion == 'failure' || github.event.workflow_run.conclusion == 'success' + runs-on: ubuntu-latest + timeout-minutes: 5 + continue-on-error: true + steps: + - name: Notify Slack + env: + GH_TOKEN: ${{ github.token }} + REPOSITORY: ${{ github.repository }} + RUN_ID: ${{ github.event.workflow_run.id }} + RUN_NUMBER: ${{ github.event.workflow_run.run_number }} + RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }} + WORKFLOW_ID: ${{ github.event.workflow_run.workflow_id }} + WORKFLOW_NAME: ${{ github.event.workflow_run.name }} + SHA: ${{ github.event.workflow_run.head_sha }} + BRANCH: ${{ github.event.workflow_run.head_branch }} + CONCLUSION: ${{ github.event.workflow_run.conclusion }} + SLACK_WEBHOOK: ${{ secrets.SLACK_RELEASE_WEBHOOK }} + run: | + set -euo pipefail + + # A reusable suite job is named "Live e2e" or ends with the exact + # " / Live e2e" suffix. Keep this aligned with the producer and gate. + suite_conclusion() { + jq -r '([.jobs[]? | select(.name == "Live e2e" or (.name | endswith(" / Live e2e"))) | .conclusion] | if any(.[]; . == "failure") then "failure" elif any(.[]; . == "success") then "success" else "none" end)' + } + + if ! current_jobs="$(gh api --paginate "/repos/${REPOSITORY}/actions/runs/${RUN_ID}/attempts/${RUN_ATTEMPT}/jobs?per_page=100")"; then + echo "::warning::Could not inspect jobs for ${WORKFLOW_NAME} run ${RUN_ID}; suppressing notification because the suite result is unknown." + exit 0 + fi + current_suite="$(printf '%s\n' "$current_jobs" | jq -s '{jobs: map(.jobs[])}' | suite_conclusion)" + startup_failure=false + if [[ "$current_suite" == none && "$WORKFLOW_NAME" == "Live E2E" && "$CONCLUSION" == failure ]]; then + current_suite=failure + startup_failure=true + elif [[ "$current_suite" == none || ("$current_suite" == failure && "$WORKFLOW_NAME" == "Release") ]]; then + # Release's generic notification covers release failures and runs + # that reused a staging result without creating a suite job. + exit 0 + fi + + prior_conclusion="" + if (( RUN_ATTEMPT > 1 )); then + if ! prior_jobs="$(gh api --paginate "/repos/${REPOSITORY}/actions/runs/${RUN_ID}/attempts/$((RUN_ATTEMPT - 1))/jobs?per_page=100")"; then + echo "::warning::Could not inspect the prior attempt for ${WORKFLOW_NAME} run ${RUN_ID}; suppressing transition notification." + exit 0 + fi + prior_conclusion="$(printf '%s\n' "$prior_jobs" | jq -s '{jobs: map(.jobs[])}' | suite_conclusion)" + [[ "$prior_conclusion" != none ]] || prior_conclusion="" + fi + + if ! history="$(gh api -X GET "/repos/${REPOSITORY}/actions/workflows/${WORKFLOW_ID}/runs" -f "branch=${BRANCH}" -F per_page=25)"; then + if [[ "$current_suite" == success && -z "$prior_conclusion" ]]; then + echo "::warning::Could not inspect recent ${WORKFLOW_NAME} history; suppressing recovery notification." + exit 0 + fi + echo "::warning::Could not inspect recent ${WORKFLOW_NAME} history; continuing with the confirmed current result." + else + candidates="$(jq -r --argjson current_number "$RUN_NUMBER" --argjson current_attempt "$RUN_ATTEMPT" '(.workflow_runs // []) | map(select((.run_number != $current_number or (.run_attempt // 1) != $current_attempt) and .status == "completed" and (.conclusion == "failure" or .conclusion == "success"))) | sort_by(.run_number, (.run_attempt // 1)) | reverse | .[] | [.id, (.run_attempt // 1), .run_number, .conclusion] | @tsv' <<< "$history")" + while IFS=$'\t' read -r candidate_id candidate_attempt candidate_number candidate_conclusion; do + [[ -z "$candidate_id" ]] && continue + if (( candidate_number > RUN_NUMBER || (candidate_number == RUN_NUMBER && candidate_attempt > RUN_ATTEMPT) )); then + relation=newer + else + relation=older + fi + if [[ "$relation" == newer || -z "$prior_conclusion" ]]; then + if ! candidate_jobs="$(gh api --paginate "/repos/${REPOSITORY}/actions/runs/${candidate_id}/attempts/${candidate_attempt}/jobs?per_page=100")"; then + echo "::warning::Could not inspect candidate run ${candidate_id}; suppressing notification because history is incomplete." + exit 0 + fi + candidate_suite="$(printf '%s\n' "$candidate_jobs" | jq -s '{jobs: map(.jobs[])}' | suite_conclusion)" + if [[ "$candidate_suite" == none && "$WORKFLOW_NAME" == "Live E2E" && "$candidate_conclusion" == failure ]]; then + candidate_suite=failure + fi + if [[ "$candidate_suite" == failure || "$candidate_suite" == success ]]; then + if [[ "$relation" == newer ]]; then + echo "A newer completed relevant suite exists; suppressing stale notification." + exit 0 + fi + prior_conclusion="$candidate_suite" + break + fi + fi + done <<< "$candidates" + fi + + if [[ "$current_suite" == success ]]; then + [[ "$prior_conclusion" == failure ]] || exit 0 + header="✅ Live E2E recovered" + text="Live E2E recovered" + detail="The previous relevant ${WORKFLOW_NAME} live suite failed; this run passed." + else + [[ "$prior_conclusion" != failure ]] || exit 0 + if [[ "$startup_failure" == true ]]; then + header="❌ Live E2E failed to start" + text="Live E2E failed to start" + detail="The Live E2E workflow failed before it produced a Live e2e job." + else + header="❌ Live E2E failed" + text="Live E2E failed" + detail="The ${WORKFLOW_NAME} live suite failed." + fi + fi + + run_url="https://github.com/${REPOSITORY}/actions/runs/${RUN_ID}" + commit_url="https://github.com/${REPOSITORY}/commit/${SHA}" + payload="$(jq -n --arg text "$text" --arg header "$header" --arg detail "$detail" --arg run_url "$run_url" --arg commit_url "$commit_url" --arg branch "$BRANCH" --arg sha "${SHA:0:7}" '{text:$text,blocks:[{type:"header",text:{type:"plain_text",text:$header,emoji:true}},{type:"section",text:{type:"mrkdwn",text:($detail + "\n*Branch:* `" + $branch + "`\n*Commit:* <" + $commit_url + "|" + $sha + ">\n*Workflow run:* <" + $run_url + "|view logs and failed tests>")}}]}')" + curl -fsSL -X POST -H 'Content-type: application/json' --data "$payload" "$SLACK_WEBHOOK" diff --git a/.github/workflows/live-e2e-suite.yml b/.github/workflows/live-e2e-suite.yml new file mode 100644 index 0000000000..1740a25051 --- /dev/null +++ b/.github/workflows/live-e2e-suite.yml @@ -0,0 +1,64 @@ +name: Live E2E Suite + +on: + workflow_call: + inputs: + ref: + description: Git ref to test + required: true + type: string + concurrency_group: + description: Concurrency group for this suite + required: false + type: string + default: live-e2e + cancel_in_progress: + description: Cancel an older run in the same group + required: false + type: boolean + default: false + secrets: + SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN: + required: true + +permissions: + contents: read + +jobs: + live-e2e: + # Consumers identify this reusable job by the exact " / Live e2e" suffix. + # Keep the suffix aligned with live-e2e-gate.yml and live-e2e-notify.yml. + name: Live e2e + runs-on: blacksmith-8vcpu-ubuntu-2404 + timeout-minutes: 30 + concurrency: + group: ${{ inputs.concurrency_group }} + cancel-in-progress: ${{ inputs.cancel_in_progress }} + env: + SUPABASE_LIVE_API_URL: https://api.supabase.green + SUPABASE_LIVE_PROJECT_NAME: supabase-cli-live + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ inputs.ref }} + persist-credentials: false + + - name: Setup + uses: ./.github/actions/setup + + - name: Docker preflight + run: docker info + + # Consumers also use this exact step name when deciding whether a suite ran. + - name: Run live e2e + timeout-minutes: 20 + env: + SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} + run: pnpm run test:live + + - name: Cleanup leftover projects + if: always() + env: + SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} + run: bash apps/cli/scripts/sweep-live-projects.sh "supabase-cli-live-${GITHUB_RUN_ID}-" diff --git a/.github/workflows/live-e2e.yml b/.github/workflows/live-e2e.yml index 180ca71787..3817b9eff1 100644 --- a/.github/workflows/live-e2e.yml +++ b/.github/workflows/live-e2e.yml @@ -1,150 +1,26 @@ name: Live E2E -# Live e2e suite. Runs the collocated `apps/cli` tests against the real staging -# Management API + Docker bundler, then invokes deployed functions over HTTP. -# -# Non-blocking by construction: this is a standalone workflow, NOT part of the -# required-checks set, and it never runs on the default PR path of test.yml. -# -# Triggers: -# - workflow_dispatch — manual run. The Actions UI branch picker selects the -# ref (github.ref), always a same-repo branch. We deliberately take NO -# free-form `ref` input: that would let a manual run check out arbitrary -# (e.g. external PR) code while the staging token is in the job env. -# - schedule (daily) — exercises the `@beta` channel. `develop` is the default -# branch AND the beta release source, so a scheduled run checks it out and -# builds from source. The `gate` job skips the run unless the published -# `supabase@beta` version changed since the last green run (an actions/cache -# marker keyed on the version), so we only spend a staging project when there -# is actually a new beta to test. -# -# Secrets: workflow_dispatch and schedule both run on trusted same-repo refs, so -# the staging token is never exposed to fork code. on: + push: + branches: [develop] workflow_dispatch: schedule: - # Daily, offset from other scheduled workflows. Cron timing is best-effort. - cron: "23 6 * * *" permissions: contents: read jobs: - # Decide whether to run. Manual dispatch always runs. Scheduled runs only fire - # when the latest published `@beta` is newer than the last one we tested green. - gate: - name: Gate (newer @beta?) - runs-on: ubuntu-latest - outputs: - should_run: ${{ steps.decide.outputs.should_run }} - version: ${{ steps.ver.outputs.version }} - steps: - - name: Resolve latest @beta version - id: ver - run: | - set -euo pipefail - version="$(npm view supabase@beta version)" - # Validate the shape before it becomes a cache key (defense-in-depth - # against a garbage/poisoned registry value). - if [[ ! "$version" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.]+)?$ ]]; then - echo "::error::unexpected supabase@beta version '$version'"; exit 1 - fi - echo "version=$version" >> "$GITHUB_OUTPUT" - - # Marker presence == "this beta already tested green". lookup-only so we - # download nothing; the marker is written by `finalize` after a green run. - - name: Check tested marker - id: cache - if: github.event_name == 'schedule' - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: .beta-marker - key: live-e2e-beta-${{ steps.ver.outputs.version }} - lookup-only: true - - - name: Decide - id: decide - run: | - if [ "${{ github.event_name }}" != "schedule" ]; then - echo "manual dispatch -> run" - echo "should_run=true" >> "$GITHUB_OUTPUT" - elif [ "${{ steps.cache.outputs.cache-hit }}" = "true" ]; then - echo "beta ${{ steps.ver.outputs.version }} already tested green -> skip" - echo "should_run=false" >> "$GITHUB_OUTPUT" - else - echo "new beta ${{ steps.ver.outputs.version }} -> run" - echo "should_run=true" >> "$GITHUB_OUTPUT" - fi - live-e2e: - needs: gate - if: needs.gate.outputs.should_run == 'true' - name: Live e2e (${{ matrix.target }}) - runs-on: blacksmith-8vcpu-ubuntu-2404 - # Serialize a target against itself across runs. Per-job-scoped project names - # mean this is mostly belt-and-braces. - concurrency: - group: live-e2e-${{ matrix.target }}-${{ github.ref }} - cancel-in-progress: true - strategy: - # Each target is an independent green/red signal. - fail-fast: false - matrix: - # ts-legacy is the only shipped CLI shell, so it's the sole live-e2e - # target. It still shells out to the Go binary for the handful of - # commands the TS port proxies (db diff, db pull, db branch *, db - # remote *, gen keys, functions download). ts-next is a later axis. - target: - - ts-legacy - # Non-secret config is job-level; the staging token is scoped to only the two - # steps that need it (run + cleanup) so build/checkout/docker never see it. - env: - SUPABASE_LIVE_API_URL: https://api.supabase.green - SUPABASE_LIVE_PROJECT_NAME: supabase-cli-live-${{ matrix.target }} - CLI_HARNESS_TARGET: ${{ matrix.target }} - steps: - - name: Checkout - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - persist-credentials: false - - - name: Setup - uses: ./.github/actions/setup - - # Docker is a hard requirement for the --use-docker bundler cell. - - name: Docker preflight - run: docker info - - - name: Run live e2e - timeout-minutes: 20 - env: - SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} - run: pnpm run test:live - - # Backstop: delete any project this job created that survived a crash. - # The script exits non-zero (failing this step) if any delete failed. - - name: Cleanup leftover projects - if: always() - env: - SUPABASE_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} - run: bash apps/cli/scripts/sweep-live-projects.sh "supabase-cli-live-${CLI_HARNESS_TARGET}-${GITHUB_RUN_ID}-" - - # Record that this beta tested green so the next scheduled run skips it. Needs - # the whole matrix: the marker is saved only if the ts-legacy leg passed (a - # red leg leaves no marker, so the next day re-runs the same beta). - finalize: - needs: [gate, live-e2e] - if: github.event_name == 'schedule' && needs.live-e2e.result == 'success' - name: Mark @beta tested - runs-on: ubuntu-latest - steps: - - name: Write marker - run: | - mkdir -p .beta-marker - echo "${{ needs.gate.outputs.version }}" > .beta-marker/version - - - name: Save tested marker - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 - with: - path: .beta-marker - key: live-e2e-beta-${{ needs.gate.outputs.version }} + # The reusable suite requires only this explicitly forwarded secret. + name: Live e2e + uses: ./.github/workflows/live-e2e-suite.yml + with: + ref: ${{ github.sha }} + # Pushes to develop share a branch group and cancel superseded pushes. + # Manual, scheduled, and stable-gate runs use independent groups because + # they each own separate temporary projects and may run concurrently. + concurrency_group: live-e2e-${{ github.event_name == 'push' && github.ref || github.run_id }} + cancel_in_progress: ${{ github.event_name == 'push' }} + secrets: + SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} diff --git a/.github/workflows/mirror-slim-image.yml b/.github/workflows/mirror-slim-image.yml index 454dc49c71..05e829da13 100644 --- a/.github/workflows/mirror-slim-image.yml +++ b/.github/workflows/mirror-slim-image.yml @@ -1,24 +1,24 @@ name: Mirror Slim Image -# Mirrors slim service images published by supabase/slim-services from -# ghcr.io/supabase/cli/: to -# public.ecr.aws/supabase/cli/:. +# Mirrors slim service images published by supabase/slim-services from ghcr.io/supabase/cli to +# public.ecr.aws/supabase/cli. The sender polls the ECR Public destination and fails its release +# unless it resolves to the exact index digest published, so the copy must be digest-preserving +# via `regctl image copy` (byte-for-byte); do not switch to `docker buildx imagetools create`, +# which can rewrite the index and change its digest. # -# The slim-services release pipeline sends a `mirror-slim-image` -# repository_dispatch to this repo, then anonymously polls the ECR Public -# destination (15-minute timeout) and fails its release unless the destination -# resolves to the exact index digest it published. The copy must therefore be -# digest-preserving: we use `regctl image copy`, which moves the whole OCI -# index (all platform manifests and referrers) byte-for-byte. Do NOT switch -# this to `docker buildx imagetools create` — it can rewrite the index and -# change its digest, breaking the sender's verification. +# The payload arrives with whatever authority holds the dispatch token, so it is treated as +# untrusted: names are pattern-checked, source/destination are derived in +# `.github/scripts/mirror-slim-image.ts` rather than trusted from the payload, and the source +# must resolve to the claimed digest before anything is copied. # -# The payload arrives with whatever authority holds the dispatch token, so it -# is validated as untrusted input: names are pattern-checked, source and -# destination are derived here rather than trusted from the payload, and the -# source must resolve to the digest claimed by the sender before anything is -# copied. +# `upload-natives-s3` copies the same native triplets (fetched from GHCR by digest and +# re-verified) to the public-read S3 bucket in the cli-artifacts-prod account, for sandboxes that +# allow `*.amazonaws.com` but block registry blob CDNs. It runs in parallel with the ECR copy so an +# ECR Public outage or missing permission never holds back the S3 mirror, is best-effort like the +# native ECR copy, and uses the PutObject-only `slim-artifacts-publisher` role +# (infra/cli-artifacts/README.md). # +# Replay locally: `bun .github/scripts/mirror-slim-image.ts --help` # Full contract: docs/design/ecr-mirror-dispatch.md in supabase/slim-services. on: @@ -50,14 +50,22 @@ concurrency: jobs: mirror: runs-on: ubuntu-latest - # The sender's poll times out after 15 minutes; fail fast instead of - # hanging past that window. - timeout-minutes: 10 + # The sender's poll times out after 15 minutes; native copies run after the + # image copy and must still finish inside that window. + timeout-minutes: 13 permissions: contents: read packages: read id-token: write steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 + with: + bun-version-file: .bun-version + - name: Validate payload id: validate env: @@ -67,44 +75,10 @@ jobs: DIGEST: ${{ github.event.client_payload.digest || inputs.digest }} PAYLOAD_SOURCE: ${{ github.event.client_payload.source }} PAYLOAD_DESTINATION: ${{ github.event.client_payload.destination }} - run: | - set -euo pipefail - if [[ ! "$SERVICE" =~ ^[a-z][a-z0-9-]*$ ]]; then - echo "::error::invalid service name: '$SERVICE'" - exit 1 - fi - if [[ ! "$VERSION" =~ ^[A-Za-z0-9._-]+$ ]]; then - echo "::error::invalid version: '$VERSION'" - exit 1 - fi - if [[ ! "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]; then - echo "::error::invalid digest: '$DIGEST'" - exit 1 - fi - SOURCE="ghcr.io/supabase/cli/${SERVICE}:${VERSION}" - DESTINATION="public.ecr.aws/supabase/cli/${SERVICE}:${VERSION}" - # Never trust the payload's source/destination strings; require them - # to match the values derived from service + version. - if [ "$EVENT_NAME" = "repository_dispatch" ]; then - if [ "$PAYLOAD_SOURCE" != "$SOURCE" ]; then - echo "::error::payload source '$PAYLOAD_SOURCE' does not match derived '$SOURCE'" - exit 1 - fi - if [ "$PAYLOAD_DESTINATION" != "$DESTINATION" ]; then - echo "::error::payload destination '$PAYLOAD_DESTINATION' does not match derived '$DESTINATION'" - exit 1 - fi - fi - { - echo "service=$SERVICE" - echo "source=$SOURCE" - echo "destination=$DESTINATION" - echo "digest=$DIGEST" - } >> "$GITHUB_OUTPUT" + run: bun .github/scripts/mirror-slim-image.ts validate - name: Install regctl - # Installed under $RUNNER_TEMP (always writable by the job user) and - # exposed to later steps via $GITHUB_PATH. + # $RUNNER_TEMP is always writable by the job user. run: | set -euo pipefail install -d "${RUNNER_TEMP}/regctl-bin" @@ -124,15 +98,9 @@ jobs: - name: Verify source digest env: - SOURCE: ${{ steps.validate.outputs.source }} + REFERENCE: ${{ steps.validate.outputs.source }} DIGEST: ${{ steps.validate.outputs.digest }} - run: | - set -euo pipefail - SOURCE_DIGEST="$(regctl manifest head "$SOURCE")" - if [ "$SOURCE_DIGEST" != "$DIGEST" ]; then - echo "::error::source $SOURCE resolves to $SOURCE_DIGEST, expected $DIGEST" - exit 1 - fi + run: bun .github/scripts/mirror-slim-image.ts verify-digest - name: Configure aws credentials uses: aws-actions/configure-aws-credentials@cbe3b392738ccf3f987d68400dafcf4b0624a56c # v6.2.4 @@ -148,56 +116,117 @@ jobs: - name: Ensure ECR Public repository exists env: SERVICE: ${{ steps.validate.outputs.service }} - run: | - set -euo pipefail - REPO_NAME="cli/${SERVICE}" - if aws ecr-public describe-repositories \ - --repository-names "$REPO_NAME" --region us-east-1 >/dev/null 2>&1; then - echo "ECR Public repository $REPO_NAME exists" - exit 0 - fi - if CREATE_ERR="$(aws ecr-public create-repository \ - --repository-name "$REPO_NAME" --region us-east-1 2>&1 >/dev/null)"; then - echo "created ECR Public repository $REPO_NAME" - elif grep -q RepositoryAlreadyExistsException <<< "$CREATE_ERR"; then - # Concurrent run for another version of the same new service won - # the creation race; the repository exists, which is all we need. - echo "ECR Public repository $REPO_NAME was created concurrently" - else - echo "$CREATE_ERR" - echo "::error::ECR Public repository '$REPO_NAME' does not exist and this role cannot create it (missing ecr-public:CreateRepository). Create it once manually — aws ecr-public create-repository --repository-name '$REPO_NAME' --region us-east-1 — then re-run this workflow." - exit 1 - fi + run: bun .github/scripts/mirror-slim-image.ts ensure-repo - name: Mirror image env: SOURCE: ${{ steps.validate.outputs.source }} DESTINATION: ${{ steps.validate.outputs.destination }} DIGEST: ${{ steps.validate.outputs.digest }} - # Copy by digest so the copy cannot race a tag move on the source; the - # whole index, all child manifests, and any referrers move as-is. - # - # The copy runs unconditionally, with no early exit when the - # destination tag already resolves to the digest: regctl's copy is - # incremental, so a re-dispatch after a complete copy is a cheap - # verification pass, while re-running after a partial failure (root - # index pushed but referrers or digest-tags missing) completes the - # copy instead of skipping it. Re-dispatches therefore still exit - # successfully with the destination digest unchanged. - run: | - set -euo pipefail - regctl image copy --referrers --digest-tags \ - "${SOURCE%:*}@${DIGEST}" "$DESTINATION" + # Copies by digest (not tag) so it can't race a tag move on the source, and runs + # unconditionally with no early exit: regctl's copy is incremental, so re-dispatching + # after a complete copy is a cheap no-op, and after a partial failure it finishes the + # copy instead of skipping it. + run: bun .github/scripts/mirror-slim-image.ts copy-image - name: Verify destination digest env: - DESTINATION: ${{ steps.validate.outputs.destination }} + REFERENCE: ${{ steps.validate.outputs.destination }} DIGEST: ${{ steps.validate.outputs.digest }} + # ECR Public can serve the tag's previous digest (or a 404) for ~30 s after the push; + # 13 reads 10 s apart allow ~2 min. + ATTEMPTS: "13" + run: bun .github/scripts/mirror-slim-image.ts verify-digest + + - name: Mirror native artifacts + continue-on-error: true + env: + SERVICE: ${{ steps.validate.outputs.service }} + VERSION: ${{ steps.validate.outputs.version }} + EVENT_NAME: ${{ github.event_name }} + # Native tags are copied without --referrers. Failure must not fail the + # image digest the sender polls; daily audit reports native drift. + run: bun .github/scripts/mirror-slim-image.ts copy-natives + + upload-natives-s3: + # Natives only ride in repository_dispatch payloads. Nothing reaches the public bucket from a + # dispatch whose source image digest check fails. The job is skipped until the + # cli-artifacts-prod account exists and its ID is set as a repository variable. + if: ${{ github.event_name == 'repository_dispatch' && vars.CLI_ARTIFACTS_AWS_ACCOUNT_ID != '' }} + runs-on: ubuntu-latest + timeout-minutes: 15 + continue-on-error: true + permissions: + contents: read + packages: read + id-token: write + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 + with: + bun-version-file: .bun-version + + - name: Validate payload + id: validate + env: + EVENT_NAME: ${{ github.event_name }} + SERVICE: ${{ github.event.client_payload.service }} + VERSION: ${{ github.event.client_payload.version }} + DIGEST: ${{ github.event.client_payload.digest }} + PAYLOAD_SOURCE: ${{ github.event.client_payload.source }} + PAYLOAD_DESTINATION: ${{ github.event.client_payload.destination }} + run: bun .github/scripts/mirror-slim-image.ts validate + + - name: Install regctl run: | set -euo pipefail - DEST_DIGEST="$(regctl manifest head "$DESTINATION")" - if [ "$DEST_DIGEST" != "$DIGEST" ]; then - echo "::error::destination $DESTINATION resolves to $DEST_DIGEST, expected $DIGEST" - exit 1 - fi - echo "$DESTINATION resolves to $DIGEST" + install -d "${RUNNER_TEMP}/regctl-bin" + curl -fsSLo "${RUNNER_TEMP}/regctl-bin/regctl" \ + https://github.com/regclient/regclient/releases/download/v0.11.5/regctl-linux-amd64 + echo "c93aa7638749f5aaac1a8e01787321889c78f0101809bb2880343478d0ba0467 ${RUNNER_TEMP}/regctl-bin/regctl" | sha256sum -c - + chmod +x "${RUNNER_TEMP}/regctl-bin/regctl" + echo "${RUNNER_TEMP}/regctl-bin" >> "$GITHUB_PATH" + "${RUNNER_TEMP}/regctl-bin/regctl" version + + - name: Log in to ghcr.io + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Verify source digest + env: + REFERENCE: ${{ steps.validate.outputs.source }} + DIGEST: ${{ steps.validate.outputs.digest }} + run: bun .github/scripts/mirror-slim-image.ts verify-digest + + - name: Fetch and verify native artifacts + id: fetch + env: + SERVICE: ${{ steps.validate.outputs.service }} + VERSION: ${{ steps.validate.outputs.version }} + EVENT_NAME: ${{ github.event_name }} + OUTPUT_DIR: ${{ runner.temp }}/natives + run: bun .github/scripts/mirror-slim-image.ts fetch-natives + + - name: Configure aws credentials + if: steps.fetch.outputs.count != '0' + uses: aws-actions/configure-aws-credentials@cbe3b392738ccf3f987d68400dafcf4b0624a56c # v6.2.4 + with: + role-to-assume: arn:aws:iam::${{ vars.CLI_ARTIFACTS_AWS_ACCOUNT_ID }}:role/slim-artifacts-publisher + # Ties every AssumeRole and PutObject entry in CloudTrail back to one workflow run. + role-session-name: mirror-slim-image-${{ github.run_id }} + aws-region: us-east-1 + + - name: Upload native artifacts to S3 + if: steps.fetch.outputs.count != '0' + env: + SERVICE: ${{ steps.validate.outputs.service }} + VERSION: ${{ steps.validate.outputs.version }} + INPUT_DIR: ${{ runner.temp }}/natives + TARGETS: ${{ steps.fetch.outputs.targets }} + run: bun .github/scripts/mirror-slim-image.ts upload-natives-s3 diff --git a/.github/workflows/mirror-template-images.yml b/.github/workflows/mirror-template-images.yml index 7a8ce0322e..3b8cb03676 100644 --- a/.github/workflows/mirror-template-images.yml +++ b/.github/workflows/mirror-template-images.yml @@ -1,10 +1,11 @@ name: Mirror template images # Keeps the ghcr.io/ECR mirror in sync with the image versions pinned in -# apps/cli-go/pkg/config/templates/Dockerfile (the single source of truth for -# `config.Images`). When the Dockerfile changes on develop — most often via a -# merged dependabot `docker` bump — this workflow detects any tag that is not -# yet mirrored and backfills it the same way `cli-go-mirror-image.yml` does. +# apps/cli/src/shared/services/Dockerfile, which `dockerfileServiceImages` +# parses as the single source of truth for local service images. When the +# Dockerfile changes on develop — most often via a merged dependabot `docker` +# bump — this workflow detects any tag that is not yet mirrored and backfills +# it the same way `cli-go-mirror-image.yml` does. # # It runs on `push` to develop (not on the PR) on purpose: mirroring needs the # AWS role + packages:write, which a dependabot-triggered `pull_request` run @@ -22,7 +23,7 @@ on: branches: - develop paths: - - apps/cli-go/pkg/config/templates/Dockerfile + - apps/cli/src/shared/services/Dockerfile workflow_dispatch: permissions: diff --git a/.github/workflows/publish-preview-cli-packages.yml b/.github/workflows/publish-preview-cli-packages.yml index 5282341687..73e2477257 100644 --- a/.github/workflows/publish-preview-cli-packages.yml +++ b/.github/workflows/publish-preview-cli-packages.yml @@ -3,58 +3,54 @@ name: Publish Preview CLI Packages # Release-notes PRs (head ref `release-notes/*`) are markdown-only and are not # meant to produce installable preview packages. # -# Default path: ready (non-draft) PRs targeting `develop`. -# `run-ci.yml` calls this workflow for drafts and stacked / non-develop PRs. +# Opt-in by label: pkg.pr.new hosts large CLI binaries, so publish only when +# someone needs a shareable install. Add `run-preview-packages` to publish +# (and re-publish on each subsequent push while labeled); remove it to cancel +# in-progress runs via this workflow's concurrency group. on: pull_request: types: - opened - synchronize - reopened - - ready_for_review - - converted_to_draft - branches: - - develop - workflow_call: - inputs: - force: - description: Publish even when the PR is a draft (used by run-ci.yml) - type: boolean - default: false - secrets: - DF_FIREWALL_TOKEN: - required: true + - labeled + - unlabeled permissions: actions: read contents: read -# Literal prefix: called workflows inherit github.workflow from the caller -# (`run-ci`). `inputs.force` separates this call from a skipped native run on -# a develop draft. +# Unrelated label events still start a run; give them a unique group so they +# cannot cancel an in-progress publish. Removing `run-preview-packages` stays +# on the main group and cancels via cancel-in-progress. concurrency: - group: publish-preview-cli-packages.yml-${{ github.event.pull_request.number || github.head_ref }}-${{ inputs.force && 'run-ci' || 'direct' }} + group: >- + publish-preview-cli-packages.yml-${{ github.event.pull_request.number || github.ref }}${{ + ((github.event.action == 'labeled' || github.event.action == 'unlabeled') + && github.event.label.name != 'run-preview-packages' + && format('-noop-{0}', github.run_id)) + || '' + }} cancel-in-progress: true jobs: build: if: | !startsWith(github.head_ref, 'release-notes/') && - (inputs.force || github.event.pull_request.draft == false) + contains(github.event.pull_request.labels.*.name, 'run-preview-packages') && + ((github.event.action != 'labeled' && + github.event.action != 'unlabeled') || + github.event.label.name == 'run-preview-packages') name: Build preview CLI packages uses: ./.github/workflows/build-cli-artifacts.yml with: version: 0.0.0-pr.${{ github.event.pull_request.number }} - shell: legacy secrets: DF_FIREWALL_TOKEN: ${{ secrets.DF_FIREWALL_TOKEN }} publish: needs: build - if: | - !startsWith(github.head_ref, 'release-notes/') && - (inputs.force || github.event.pull_request.draft == false) && - needs.build.result == 'success' + if: needs.build.result == 'success' name: Publish preview package runs-on: blacksmith-8vcpu-ubuntu-2404 outputs: @@ -79,7 +75,7 @@ jobs: path: | packages/cli-*/bin/ dist/ - key: cli-build-${{ github.run_id }}-legacy-${{ env.PREVIEW_VERSION }}-v1 + key: cli-build-${{ github.run_id }}-${{ env.PREVIEW_VERSION }}-v1 enableCrossOsArchive: true fail-on-cache-miss: true @@ -135,10 +131,7 @@ jobs: comment: needs: publish - if: | - !startsWith(github.head_ref, 'release-notes/') && - (inputs.force || github.event.pull_request.draft == false) && - needs.publish.result == 'success' + if: needs.publish.result == 'success' name: Post preview command comment runs-on: ubuntu-latest permissions: diff --git a/.github/workflows/release-config.yml b/.github/workflows/release-config.yml index c002f39c8f..3ab2d2ac0a 100644 --- a/.github/workflows/release-config.yml +++ b/.github/workflows/release-config.yml @@ -7,16 +7,12 @@ on: paths: - "packages/config/**" - ".github/workflows/release-config.yml" - # workflow_dispatch is the manual re-cut path, mirroring the CLI's Release - # workflow. Defaults to `true` so a stray "Run workflow" click can't - # accidentally publish — operators must consciously untick this. + # workflow_dispatch is the manual re-cut path, mirroring the CLI's Release workflow. Defaults + # to `true` so a stray "Run workflow" click can't accidentally publish. # - # There is deliberately no `version` input: the publish job's registry probe - # skips versions that already exist on npm (after verifying the registry - # bytes match the reviewed artifact), so recovery from stale published - # bytes is "land a new (releasable) commit" — with no binary artifacts and a - # human approval in the loop, the CLI's cut-forward escape hatch isn't worth - # a second code path here. + # There is no `version` input: the publish job's registry probe already skips versions that + # exist on npm with matching bytes. See packages/config/AGENTS.md's recovery section for the + # published-but-untagged case. workflow_dispatch: inputs: dry_run: @@ -45,11 +41,9 @@ jobs: npm_tag: ${{ steps.plan.outputs.npm_tag }} dry_run: ${{ steps.plan.outputs.dry_run }} steps: - # semantic-release runs `git push --dry-run HEAD:` as part of - # verifyAuth even in `dry_run: true` mode, so the token must have push - # access to the protected `develop` branch. The default GITHUB_TOKEN - # doesn't, so we mint an App-installation token from the same App used - # by the CLI's release pipeline. + # semantic-release runs `git push --dry-run HEAD:` as part of verifyAuth even in + # `dry_run: true` mode, so the token needs push access to the protected develop branch — + # the default GITHUB_TOKEN doesn't have it. - id: app-token uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 with: @@ -57,11 +51,9 @@ jobs: private-key: ${{ secrets.GH_APP_PRIVATE_KEY }} permission-contents: write - # `persist-credentials: false` is required: otherwise checkout caches the - # default GITHUB_TOKEN as an `http.extraheader` in git config, and that - # Authorization header overrides the App token semantic-release puts in - # the push URL — making the dry-push identify as `github-actions[bot]` - # and get rejected by branch protection. + # `persist-credentials: false` is required: otherwise checkout caches the default + # GITHUB_TOKEN in git config, and its Authorization header overrides the App token + # semantic-release puts in the push URL, making the dry-push get rejected by branch protection. - uses: useblacksmith/checkout@6fd481652155169ed4d2f25ebaf97464f685175f # v1 with: fetch-depth: 0 @@ -101,20 +93,16 @@ jobs: trap 'echo "::${resume_token}::"' EXIT pnpm exec bun packages/config/scripts/release-plan.ts --notes-out "$RUNNER_TEMP/config-release-notes.md" - # The build, gate, and pack steps also run on private-blocked pushes - # (should_release=false, version set): if `private` were ever flipped - # back on, every config push would still rehearse the plan half of the - # release train while the publish half stays parked. + # Build/gate/pack also run on private-blocked pushes (should_release=false, version set): + # if `private` is ever flipped back on, the plan half of the release train is already + # rehearsed on every config push. - name: Build @supabase/config if: steps.plan.outputs.version != '' run: pnpm exec turbo run @supabase/config#build - # Pack the exact tarball the approver's evidence (the gate summary - # below) describes. The publish job publishes THIS artifact rather than - # rebuilding: builds are not byte-reproducible across jobs (see - # release-shared.yml's brew/scoop cache-key comments for how that bit - # once before), and a rebuild would mean the approved bytes and the - # published bytes can differ. + # Packs the exact tarball the approver's evidence (the gate summary below) describes, + # rather than rebuilding: builds aren't byte-reproducible across jobs, so a rebuild could + # let the approved bytes and published bytes differ. - name: Pack the release tarball if: steps.plan.outputs.version != '' env: @@ -127,11 +115,9 @@ jobs: pnpm pack --pack-destination "$RUNNER_TEMP/config-release-artifact" cp "$RUNNER_TEMP/config-release-notes.md" "$RUNNER_TEMP/config-release-artifact/" - # The gate diffs the declarations INSIDE the packed tarball (not the - # dist/ build directory it was packed from), so the approver's evidence - # is generated from the same bytes the publish job ships — a packlist - # regression that drops .d.ts files from the tarball fails here instead - # of shipping a surface the approver never saw. + # Diffs the declarations inside the packed tarball (not the dist/ directory it was packed + # from), so a packlist regression that drops .d.ts files fails here instead of shipping a + # surface the approver never saw. - name: Run type-surface release gate against the packed tarball if: steps.plan.outputs.version != '' env: @@ -153,13 +139,10 @@ jobs: if-no-files-found: error retention-days: 7 - # The `config-release` environment's required-reviewers rule is repo - # configuration, not code: an environment referenced by a workflow is - # auto-created WITHOUT protection rules, in which case the publish job - # would run straight through unreviewed. Fail closed here — before a - # real (non-dry) release can reach the publish job — if the rule is - # missing or unreadable. Private-blocked rehearsals (should_release - # false) are unaffected — this only gates real releases. + # An environment referenced by a workflow is auto-created without protection rules, so if + # `config-release`'s required-reviewers rule (repo config, not code) goes missing, the + # publish job would run straight through unreviewed. Fail closed here before a real + # (non-dry) release can reach it; private-blocked rehearsals are unaffected. - name: Assert the release approval gate is armed if: steps.plan.outputs.should_release == 'true' && steps.plan.outputs.dry_run != 'true' env: @@ -194,26 +177,21 @@ jobs: name: Publish needs: plan if: needs.plan.outputs.should_release == 'true' && needs.plan.outputs.dry_run != 'true' - # npm provenance verification rejects non-GitHub-hosted runners with - # E422 ("Unsupported GitHub Actions runner environment: self-hosted"). - # Blacksmith runners count as self-hosted from sigstore's POV, so the - # publish job must stay on a github-hosted runner. The job is short and - # not compute-bound, so the wall-clock cost is negligible. + # npm provenance verification rejects non-GitHub-hosted runners with E422; Blacksmith runners + # count as self-hosted from sigstore's POV, so this job must stay on a github-hosted runner + # (it's short and not compute-bound, so the cost is negligible). runs-on: ubuntu-latest timeout-minutes: 30 environment: name: config-release - # This environment must be configured with required reviewers in repo - # settings (asserted by the plan job above). The approver reviews the - # plan job's step summary (release notes + type-surface gate diff) - # before approving — that approval IS the hard semver gate, and the - # tarball published below is byte-identical to the one that evidence - # was generated from. + # Configured with required reviewers in repo settings (asserted by the plan job above). The + # approver reviews the plan job's step summary (release notes + type-surface gate diff) + # before approving — that approval is the hard semver gate, and the tarball published below + # is byte-identical to what that evidence was generated from. url: https://www.npmjs.com/package/@supabase/config/v/${{ needs.plan.outputs.version }} - # OIDC trusted publishing + provenance — same as release-shared.yml; no - # NPM_TOKEN anywhere. This job deliberately runs NO dependency install and - # NO build: the only repo code it executes is this workflow file, keeping - # arbitrary package code away from the job that holds id-token: write. + # OIDC trusted publishing + provenance — same as release-shared.yml; no NPM_TOKEN anywhere. + # This job runs no dependency install and no build: the only repo code it executes is this + # workflow file, keeping arbitrary package code away from the job that holds id-token: write. permissions: contents: write id-token: write @@ -237,9 +215,8 @@ jobs: persist-credentials: true token: ${{ steps.app-token.outputs.token }} - # npm only — no `pnpm install`, no workspace toolchain. npm ≥ 11.5.1 is - # required for OIDC trusted publishing, newer than the runner image's - # system npm, and node 24 bundles it. + # npm only — no `pnpm install`, no workspace toolchain. npm ≥ 11.5.1 (required for OIDC + # trusted publishing) is newer than the runner image's system npm; node 24 bundles it. - name: Install node uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: @@ -267,12 +244,9 @@ jobs: exit 1 fi - # Idempotent, mirroring publish.ts's registry-probe intent: a re-run - # after a post-publish failure must not die on EPUBLISHCONFLICT — but - # the skip is only safe when the registry's bytes ARE the reviewed - # artifact, since the tag push below would otherwise bless foreign - # bytes (e.g. a previous run's tag push failed, new commits landed, - # and a fresh plan recomputed the same version from a newer tree). + # Idempotent, mirroring publish.ts's registry-probe intent: a re-run after a post-publish + # failure must not die on EPUBLISHCONFLICT. The skip is only safe when the registry's bytes + # match the reviewed artifact, since the tag push below would otherwise bless foreign bytes. - name: Publish to npm working-directory: ${{ runner.temp }}/config-release run: | @@ -285,7 +259,8 @@ jobs: echo "@supabase/config@${VERSION} already exists on npm but does not match the reviewed tarball:" >&2 echo " registry: ${registry_integrity}" >&2 echo " reviewed: ${reviewed_integrity}" >&2 - echo "Refusing to tag a commit for bytes this run never reviewed — cut forward by landing a new releasable commit." >&2 + echo "Refusing to tag a commit for bytes this run never reviewed." >&2 + echo "See packages/config/AGENTS.md's recovery section if config-v${VERSION} is also missing from origin." >&2 exit 1 fi echo "@supabase/config@${VERSION} already on npm with matching integrity; skipping publish." @@ -298,18 +273,30 @@ jobs: npm publish "./${tarball}" --ignore-scripts --provenance --tag "${NPM_TAG}" fi - # "Published successfully" for the notification jobs below means - # REGISTRY-VISIBLE with the reviewed bytes: probe npm until the version - # resolves (the registry can lag a publish by a few seconds) and its - # integrity matches the reviewed tarball. Runs before the tag push so a - # tag is never blessed for bytes this run couldn't confirm on the - # registry; the dist-tag assertion deliberately lives AFTER the tag push - # so a dist-tag propagation hiccup can't strand the release live on npm - # with origin untagged. A transient failure here is safe to re-run — the - # publish step's registry probe skips the republish and this check - # repeats — though a re-run of this job re-arms the config-release - # approval gate and costs a second human approval — hence the generous - # (~2 minute) visibility budget. + - name: Configure git for release pushes + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + # Runs ahead of the verification steps: once npm accepts the bytes, the tag must land + # regardless of how the probes below fare, since release-plan.ts derives the next version + # from the last config-v* tag, never from npm (recovery: packages/config/AGENTS.md). Idempotent. + - name: Push version tag + run: | + set -euo pipefail + tag="config-v${VERSION}" + if git ls-remote --tags origin "refs/tags/${tag}" | grep -q .; then + echo "Tag ${tag} already on origin; skipping push." + else + git tag -a "${tag}" -m "Release ${tag}" + git push origin "${tag}" + fi + + # "Published successfully" for the notification jobs means registry-visible with the + # reviewed bytes: probe npm until the version resolves and its integrity matches. Runs after + # the tag push, so a slow registry read here can fail without stranding the release + # published-but-untagged. A re-run of this job re-arms the approval gate and costs a second + # human approval, hence the generous visibility budget. - name: Verify the release is live on npm working-directory: ${{ runner.temp }}/config-release run: | @@ -318,7 +305,7 @@ jobs: reviewed_integrity="sha512-$(openssl dgst -sha512 -binary "${tarball}" | base64 -w0)" npm_err="$RUNNER_TEMP/npm-view-err.log" registry_integrity="" - for delay in 0 2 3 5 8 13 21 30 30; do + for delay in 0 2 3 5 8 13 21 30 30 30 60 60 60 60 60 60 60 60; do sleep "$delay" if registry_integrity="$(npm view --prefer-online "@supabase/config@${VERSION}" dist.integrity 2>"$npm_err")" \ && [[ -n "$registry_integrity" ]]; then @@ -342,33 +329,9 @@ jobs: fi echo "@supabase/config@${VERSION} live on npm with the reviewed bytes." - - name: Configure git for release pushes - run: | - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - - # Push the tag to origin as soon as npm has the bytes, before any - # downstream step that can fail. Without this, a failure in the GH - # release step leaves origin with no tag for the version that is now - # live on npm — and a subsequent plan would recompute the same version - # against stale bytes. Idempotent: skips push if the tag is already on - # origin (e.g. a re-run of a job that previously got past this step). - - name: Push version tag - run: | - set -euo pipefail - tag="config-v${VERSION}" - if git ls-remote --tags origin "refs/tags/${tag}" | grep -q .; then - echo "Tag ${tag} already on origin; skipping push." - else - git tag -a "${tag}" -m "Release ${tag}" - git push origin "${tag}" - fi - - # Asserted after the tag push on purpose: npm already holds the reviewed - # bytes (verified above), so a dist-tag propagation hiccup must not - # strand the release npm-published but origin-untagged. Retries until - # the tag points at THIS version — a stale packument still echoing the - # previous version is a retryable state, not a terminal mismatch. + # The tag was already pushed above, so this and the probe above are both post-tag + # confirmations — this one waits for the dist-tag pointer specifically, since a stale + # packument (tag not yet updated) is retryable, not a terminal mismatch. - name: Verify the npm dist-tag run: | set -euo pipefail @@ -406,13 +369,10 @@ jobs: # become the repo's "latest" release. make_latest: "false" - # Pages the release Slack channel the moment a real run arms the - # config-release approval gate. This job only needs `plan`, so it runs in - # parallel with the publish job's `waiting` state — the ping and the pending - # deployment appear together. The approval itself stays on GitHub: the run - # page hosts the Approve button and the plan job's evidence summary; the - # webhook is one-way and cannot host an interactive approval. Nothing - # depends on this job, so a Slack/webhook failure can't block the release. + # Pages Slack the moment a real run arms the config-release approval gate. Runs in parallel + # with the publish job's `waiting` state (needs only `plan`), so the ping and the pending + # deployment appear together — the approval itself stays on GitHub, since the webhook is + # one-way and can't host an interactive approval. notify-slack-approval: name: Notify Slack (approval needed) needs: plan @@ -426,12 +386,9 @@ jobs: secrets: SLACK_RELEASE_WEBHOOK: ${{ secrets.SLACK_RELEASE_WEBHOOK }} - # Posts once the publish job has verified the release is registry-visible - # with the reviewed bytes. No `if:` needed: the implicit success() gate - # means this only runs when plan and publish both succeeded, and publish - # itself only runs for real (non-dry) releases — dry runs and no-release - # pushes skip publish, which skips this too. Nothing depends on this job, - # so a Slack/webhook failure can't affect the already-completed release. + # Posts once publish has verified the release is registry-visible with the reviewed bytes. No + # `if:` needed: the implicit success() gate means this only runs when plan and publish both + # succeeded, and publish itself only runs for real releases. notify-slack: name: Notify Slack needs: [plan, publish] @@ -445,14 +402,10 @@ jobs: secrets: SLACK_RELEASE_WEBHOOK: ${{ secrets.SLACK_RELEASE_WEBHOOK }} - # Distinguishes "a reviewer rejected the pending deployment" from a real - # pipeline failure before paging the channel: a rejection marks the publish - # job failed, and announcing that as a broken release would page people - # about a deliberate decision. The run's approvals record is the only place - # the distinction is visible from inside the workflow. The dry-run guard - # reads the dispatch input directly rather than plan's output, so a plan - # job that dies before recording dry_run still can't page for an - # operator-watched dry run. + # Distinguishes a reviewer's rejection from a real pipeline failure before paging the channel — + # a rejection marks the publish job failed, and announcing that as a broken release would page + # people about an intentional call. The dry-run guard reads the dispatch input directly, so a + # plan job that dies before recording `dry_run` still can't page for an operator-watched run. classify-failure: name: Classify failure needs: [plan, publish] @@ -493,14 +446,10 @@ jobs: echo "status=failure" >> "$GITHUB_OUTPUT" fi - # Reports a failed (or reviewer-declined) release. `failure()` on the - # classify job evaluates against its `needs` chain, so this pair fires - # whenever `plan` or `publish` fails — including an approval rejection — - # but stays quiet for dry runs and no-release pushes (skipped needs don't - # count as failures). When `plan` fails its outputs are empty, so the - # message falls back to the workflow run link as the actionable detail. - # Fails open: if the classifier itself breaks, the page still goes out as a - # plain failure (its status output is empty, so the expression falls back). + # Reports a failed (or reviewer-declined) release. `failure()` on the classify job evaluates + # against its `needs` chain, so this pair fires whenever `plan` or `publish` fails — including + # an approval rejection — but stays quiet for dry runs and no-release pushes. Fails open: if + # the classifier itself breaks, the page still goes out as a plain failure. notify-slack-failure: name: Notify Slack (failure) needs: [plan, publish, classify-failure] diff --git a/.github/workflows/release-shared.yml b/.github/workflows/release-shared.yml index 3ad7fec72e..57b1d3e8a5 100644 --- a/.github/workflows/release-shared.yml +++ b/.github/workflows/release-shared.yml @@ -7,10 +7,6 @@ on: description: npm package version to publish required: true type: string - shell: - description: CLI shell to package as the shipped supabase binary - required: true - type: string npm_tag: description: npm dist-tag to publish under required: true @@ -59,13 +55,14 @@ on: required: false LINEAR_CLI_BETA_RELEASE_ACCESS_KEY: required: false + SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN: + required: false jobs: build-blacksmith: name: Build CLI artifacts (Blacksmith) uses: ./.github/workflows/build-cli-artifacts.yml with: version: ${{ inputs.version }} - shell: ${{ inputs.shell }} secrets: SENTRY_DSN: ${{ secrets.SENTRY_DSN }} POSTHOG_API_KEY: ${{ secrets.POSTHOG_API_KEY }} @@ -77,7 +74,6 @@ jobs: uses: ./.github/workflows/build-cli-artifacts.yml with: version: ${{ inputs.version }} - shell: ${{ inputs.shell }} runner: large-linux-x86 cache_key_suffix: -github timeout_minutes: 45 @@ -119,16 +115,14 @@ jobs: path: | packages/cli-*/bin/ dist/ - key: cli-build-${{ github.run_id }}-${{ inputs.shell }}-${{ inputs.version }}-v1 + key: cli-build-${{ github.run_id }}-${{ inputs.version }}-v1 enableCrossOsArchive: true fail-on-cache-miss: true - # Docker's classic image store keeps a single platform manifest per - # tag, so pulling `alpine:3.21` for amd64 and again for arm64 leaves - # only the most recent one in the local store — and `docker save` - # captures only that one. Enabling the containerd snapshotter lets - # the store hold multi-platform manifests under a single tag, which - # is what makes the cache step below work for both arches. + # Docker's classic image store keeps a single platform manifest per tag, so pulling + # `alpine:3.21` for amd64 then arm64 leaves only the most recent in the local store, and + # `docker save` captures only that one. The containerd snapshotter lets the store hold + # multi-platform manifests under a single tag instead. - name: Enable containerd image store if: runner.os == 'Linux' run: | @@ -137,17 +131,12 @@ jobs: sudo systemctl restart docker docker info --format '{{.DriverStatus}}' - # The host's binfmt_misc must be mounted BEFORE installing QEMU. On cold - # Blacksmith VMs it is not mounted by default; the privileged - # tonistiigi/binfmt installer that setup-qemu-action runs then registers - # the qemu interpreters inside its own mount namespace — it prints - # "installing: arm64 OK" but the handlers vanish when the container exits, - # so the host kernel never gains arm64 emulation and every - # `docker run --platform linux/arm64` dies with "exec format error". - # Warm/reused VMs already had it mounted, which is why this only failed - # intermittently (on cache-miss, i.e. cold-VM, runs). Mount it on the host - # after the docker restart so the registration lands in the host kernel - # and persists through to the smoke tests. + # The host's binfmt_misc must be mounted before installing QEMU. On cold Blacksmith VMs it + # isn't mounted by default, so setup-qemu-action's privileged installer registers the qemu + # interpreters inside its own mount namespace — they vanish when the container exits, and + # `docker run --platform linux/arm64` fails with "exec format error" (only on cold-VM runs, + # since warm VMs already have it mounted). Mount it on the host after the docker restart so + # the registration lands in the host kernel. - name: Ensure binfmt_misc is mounted on the host if: runner.os == 'Linux' run: | @@ -160,7 +149,7 @@ jobs: - name: Setup QEMU for cross-platform Docker if: runner.os == 'Linux' - uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4.2.0 + uses: docker/setup-qemu-action@1f40c72289eff860ee54a304f1438e3cff362e0a # v4.3.0 - name: Verify linux/arm64 emulation is registered if: runner.os == 'Linux' @@ -171,11 +160,10 @@ jobs: # the host kernel. Reads binfmt_misc directly — no image pull, no network. grep -q enabled /proc/sys/fs/binfmt_misc/qemu-aarch64 - # Cache the smoke-test base images across runs. Without this, eight - # parallel `docker run` calls in smoke-test-linux.ts race on first-time - # pulls and surface as docker exit 125 ("daemon could not start the - # container") on a subset of jobs. Bump the `-v` suffix to invalidate - # the cache when the image list or tags change. + # Caches the smoke-test base images across runs — without it, eight parallel `docker run` + # calls in smoke-test-linux.ts race on first-time pulls and fail with docker exit 125 + # ("daemon could not start the container"). Bump the `-v` suffix to invalidate when the + # image list or tags change. - name: Cache smoke-test docker images if: runner.os == 'Linux' id: smoke-docker-cache @@ -223,9 +211,8 @@ jobs: run: chmod +x packages/cli-*/bin/supabase || true - name: Run smoke tests - # Force bash so ${VERSION}/${NPM_TAG} expand identically across the - # ubuntu/macos/windows matrix — windows-latest defaults to pwsh, which - # treats those as empty PowerShell variables (env vars are `$env:VAR`). + # Force bash so ${VERSION}/${NPM_TAG} expand identically across the ubuntu/macos/windows + # matrix — windows-latest defaults to pwsh, which treats those as empty PowerShell variables. shell: bash run: pnpm run test:smoke -- --version "${VERSION}" --tag "${NPM_TAG}" working-directory: apps/cli @@ -255,7 +242,7 @@ jobs: path: | packages/cli-*/bin/ dist/ - key: cli-build-${{ github.run_id }}-${{ inputs.shell }}-${{ inputs.version }}-github-v1 + key: cli-build-${{ github.run_id }}-${{ inputs.version }}-github-v1 enableCrossOsArchive: true fail-on-cache-miss: true @@ -267,17 +254,25 @@ jobs: run: pnpm run test:smoke -- --version "${VERSION}" --tag "${NPM_TAG}" working-directory: apps/cli + live-e2e-gate: + name: Verify live e2e before stable publication + if: ${{ inputs.channel == 'stable' && !inputs.dry_run }} + uses: ./.github/workflows/live-e2e-gate.yml + with: + sha: ${{ github.sha }} + secrets: + SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} + publish: needs: - build-github - smoke-test - smoke-test-macos - if: ${{ !inputs.dry_run }} - # npm provenance verification rejects non-GitHub-hosted runners with - # E422 ("Unsupported GitHub Actions runner environment: self-hosted"). - # Blacksmith runners count as self-hosted from sigstore's POV, so the - # publish job must stay on a github-hosted runner. The job is short - # and not compute-bound, so the wall-clock cost is negligible. + - live-e2e-gate + if: ${{ !cancelled() && !inputs.dry_run && needs.build-github.result == 'success' && needs.smoke-test.result == 'success' && needs.smoke-test-macos.result == 'success' && (inputs.channel != 'stable' || needs.live-e2e-gate.result == 'success') }} + # npm provenance verification rejects non-GitHub-hosted runners with E422; Blacksmith runners + # count as self-hosted from sigstore's POV, so this job must stay on a github-hosted runner + # (it's short and not compute-bound, so the cost is negligible). runs-on: ubuntu-latest timeout-minutes: 45 env: @@ -317,7 +312,7 @@ jobs: path: | packages/cli-*/bin/ dist/ - key: cli-build-${{ github.run_id }}-${{ inputs.shell }}-${{ inputs.version }}-github-v1 + key: cli-build-${{ github.run_id }}-${{ inputs.version }}-github-v1 enableCrossOsArchive: true fail-on-cache-miss: true @@ -332,13 +327,10 @@ jobs: git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - # Push the version tag to origin as soon as npm has the bytes, before any - # downstream step that can fail. Without this, a failure in the GH-release - # step (or anything after) leaves origin with no tag for the version that - # is now live on npm — and `semantic-release --dry-run` on the next push - # will keep recomputing the same version, causing the publish job to no-op - # against stale bytes. Idempotent: skips push if the tag is already on - # origin (e.g. a re-run of a job that previously got past this step). + # Pushes the tag as soon as npm has the bytes, before any downstream step that can fail — + # otherwise a later failure leaves origin untagged for a version already live on npm, and + # `semantic-release --dry-run` on the next push keeps recomputing the same version. + # Idempotent: skips if the tag is already on origin. - name: Push version tag run: | set -euo pipefail @@ -350,37 +342,32 @@ jobs: git push origin "${tag}" fi - # semantic-release matches tags against branch.channel using git-notes - # under refs/notes/semantic-release-. Without this note, the develop - # plan job recomputes the same prerelease version every push (see - # docs/ci-beta-tags-investigation.md). Stable releases on main don't - # need this — main's branch config has no explicit channel. + # semantic-release matches tags against branch.channel using git-notes under + # refs/notes/semantic-release-; without this note, the develop plan job recomputes + # the same prerelease version every push (see docs/ci-beta-tags-investigation.md). Stable + # releases on main don't need this, since main's branch config has no explicit channel. # - # The note must be attached to the underlying commit, not the annotated - # tag object created by "Push version tag" above. semantic-release reads - # notes via `git log --no-walk --tags=* --notes=...`, which looks for - # notes on commits; a note on the tag object is invisible to it and - # leaves the tag with channels:[null], so the prerelease arm of - # getLastRelease's filter rejects it and the plan falls back to the - # last stable. Hence the `${tag}^{commit}` deref. + # The note must attach to the underlying commit, not the annotated tag object created by + # "Push version tag" above — semantic-release reads notes via `git log --no-walk + # --tags=* --notes=...`, which looks at commits, so a note on the tag object leaves the tag + # with channels:[null] and the plan falls back to the last stable. Hence the + # `${tag}^{commit}` deref. - name: Push channel note for prerelease tags if: ${{ inputs.prerelease }} run: | set -euo pipefail tag="v${VERSION}" - channel="${CHANNEL}" note_ref="refs/notes/semantic-release-${tag}" if git ls-remote origin "${note_ref}" | grep -q .; then echo "Channel note ${note_ref} already on origin; skipping push." else - git notes --ref="${note_ref}" add -f -m "{\"channels\":[\"${channel}\"]}" "${tag}^{commit}" + git notes --ref="${note_ref}" add -f -m "{\"channels\":[\"${CHANNEL}\"]}" "${tag}^{commit}" git push origin "${note_ref}" fi - # goreleaser used to publish both supabase___.tar.gz and - # supabase__.tar.gz; setup-cli, install scripts, docs and various - # third-party tools point at releases/latest/download/.tar.gz. - # Re-publish the alias so those callers keep working. See supabase/cli#5257. + # setup-cli, install scripts, docs, and other tools reference + # releases/latest/download/.tar.gz, so republish that alias alongside the + # versioned tarball. - name: Create unversioned tarball aliases run: | set -euo pipefail @@ -426,7 +413,7 @@ jobs: - name: Sync stable release to Linear if: ${{ inputs.channel == 'stable' && env.LINEAR_CLI_STABLE_RELEASE_ACCESS_KEY != '' }} - uses: linear/linear-release-action@3f31fcf14c110cc53579fcc3575a26d469c413b4 # v0 + uses: linear/linear-release-action@53ad0f863963e7f8e270fba18426bbb55ef55384 # v0 with: access_key: ${{ env.LINEAR_CLI_STABLE_RELEASE_ACCESS_KEY }} version: v${{ inputs.version }} @@ -437,7 +424,7 @@ jobs: - name: Sync beta release to Linear if: ${{ inputs.channel == 'beta' && env.LINEAR_CLI_BETA_RELEASE_ACCESS_KEY != '' }} - uses: linear/linear-release-action@3f31fcf14c110cc53579fcc3575a26d469c413b4 # v0 + uses: linear/linear-release-action@53ad0f863963e7f8e270fba18426bbb55ef55384 # v0 with: access_key: ${{ env.LINEAR_CLI_BETA_RELEASE_ACCESS_KEY }} version: v${{ inputs.version }} @@ -459,11 +446,10 @@ jobs: secrets: DF_FIREWALL_TOKEN: ${{ secrets.DF_FIREWALL_TOKEN }} - # Once the raw semantic-release block is in the release body, ask Claude to - # rewrite it into user-centric notes and open a PR for human approval. Stable - # releases only on this path — prereleases keep the raw body. Non-blocking so - # an LLM hiccup never gates a published release; reviewers can propose beta - # notes manually from the Actions tab (workflow_dispatch). + # Once the raw semantic-release block is in the release body, asks Claude to rewrite it into + # user-centric notes and open a PR for human approval. Stable only — prereleases keep the raw + # body. Non-blocking so an LLM hiccup never gates a published release; reviewers can propose + # beta notes manually via workflow_dispatch. propose-release-notes: uses: ./.github/workflows/propose-release-notes.yml needs: backfill-release-notes @@ -497,19 +483,17 @@ jobs: with: dependency-firewall-token: ${{ secrets.DF_FIREWALL_TOKEN }} - # Must restore the github-hosted build (-github-v1), the same artifacts - # the publish job uploads to the GitHub Release. The Bun-compiled binaries - # are not byte-for-byte reproducible across the blacksmith and github - # builds, so the blacksmith dist/checksums.txt does not match the released - # tarballs. Reading it here produced a formula whose sha256 rejected the - # downloaded archive ("Formula reports different checksum"). + # Restores the github-hosted build (-github-v1) — the same artifacts the publish job + # uploads to the GitHub Release. Bun-compiled binaries aren't byte-for-byte reproducible + # across the blacksmith and github builds, so the blacksmith checksums.txt doesn't match + # the released tarballs, and a formula built from it rejects the downloaded archive. - name: Restore build artifacts cache uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | packages/cli-*/bin/ dist/ - key: cli-build-${{ github.run_id }}-${{ inputs.shell }}-${{ inputs.version }}-github-v1 + key: cli-build-${{ github.run_id }}-${{ inputs.version }}-github-v1 enableCrossOsArchive: true fail-on-cache-miss: true @@ -560,19 +544,17 @@ jobs: with: dependency-firewall-token: ${{ secrets.DF_FIREWALL_TOKEN }} - # Must restore the github-hosted build (-github-v1), the same artifacts - # the publish job uploads to the GitHub Release. The Bun-compiled binaries - # are not byte-for-byte reproducible across the blacksmith and github - # builds, so the blacksmith dist/checksums.txt does not match the released - # tarballs. Reading it here would produce a manifest whose hash rejects the - # downloaded archive. + # Restores the github-hosted build (-github-v1) — the same artifacts the publish job + # uploads to the GitHub Release. Bun-compiled binaries aren't byte-for-byte reproducible + # across the blacksmith and github builds, so the blacksmith checksums.txt doesn't match + # the released tarballs, and a manifest built from it rejects the downloaded archive. - name: Restore build artifacts cache uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: | packages/cli-*/bin/ dist/ - key: cli-build-${{ github.run_id }}-${{ inputs.shell }}-${{ inputs.version }}-github-v1 + key: cli-build-${{ github.run_id }}-${{ inputs.version }}-github-v1 enableCrossOsArchive: true fail-on-cache-miss: true @@ -602,18 +584,12 @@ jobs: GH_TOKEN: ${{ steps.app-token.outputs.token }} GITHUB_TOKEN: ${{ steps.app-token.outputs.token }} - # Post-publish smoke test for the `supabase/setup-cli` GitHub Action against - # the just-released CLI. Runs last and intentionally does not gate - # publish-homebrew / publish-scoop — by the time the smoke runs, the npm - # package and GitHub release are already live and the brew/scoop pushes - # have either succeeded or skipped, so a setup-cli regression surfaces as - # a red post-release signal without holding back the rest of the channel. - # - # Depends on the publish jobs only via `needs` for ordering; the `if` - # uses `always() && needs.publish.result == 'success'` so the smoke still - # runs when publish-homebrew / publish-scoop are skipped or fail. - # The reusable workflow can also be dispatched manually against any - # already-published version when debugging setup-cli regressions. + # Post-publish smoke test for the `supabase/setup-cli` GitHub Action against the just-released + # CLI. Runs last and intentionally doesn't gate publish-homebrew/publish-scoop — by the time it + # runs, the npm package and GitHub release are already live, so a setup-cli regression surfaces + # as a red post-release signal without holding back the rest of the channel. `needs` is for + # ordering only; `always() && needs.publish.result == 'success'` lets it run even when + # publish-homebrew/publish-scoop are skipped or fail. setup-cli-smoke: needs: [publish, publish-homebrew, publish-scoop] if: ${{ always() && !inputs.dry_run && needs.publish.result == 'success' }} @@ -621,16 +597,11 @@ jobs: with: version: ${{ inputs.version }} - # Post-publish end-to-end check that the Homebrew tap, Scoop bucket, and the - # curl|bash install script actually install the just-released CLI. brew/scoop - # verify the published checksum against the downloaded tarball, so this is the - # signal that would have caught CLI v2.107.0 (mismatched brew/scoop sha256s). - # - # Only runs when brew/scoop were published (beta/stable) and both pushes - # succeeded. A channel that skipped brew/scoop entirely is covered by the GitHub - # Release download path in setup-cli-smoke. Like setup-cli-smoke, it runs last - # and does not gate the rest of the channel: by the time it runs the manifests - # are already live, so a failure surfaces as a red post-release signal. + # Post-publish check that the Homebrew tap, Scoop bucket, and curl|bash install script + # actually install the just-released CLI — brew/scoop verify the published checksum against + # the downloaded tarball, catching a mismatch like CLI v2.107.0. Only runs when brew/scoop were + # published and both pushes succeeded; a skipped channel is covered by the GitHub Release + # download path in setup-cli-smoke. Runs last and doesn't gate the rest of the channel. verify-install-channels: needs: [publish, publish-homebrew, publish-scoop] if: ${{ always() && !inputs.dry_run && inputs.publish_brew_scoop && needs.publish-homebrew.result == 'success' && needs.publish-scoop.result == 'success' }} diff --git a/.github/workflows/release-smoke-test.yml b/.github/workflows/release-smoke-test.yml index a21df3e0cf..a9581a9e1e 100644 --- a/.github/workflows/release-smoke-test.yml +++ b/.github/workflows/release-smoke-test.yml @@ -8,13 +8,6 @@ on: required: false type: string default: 0.0.0-smoke - shell: - description: CLI shell to package as the shipped supabase binary - required: false - type: choice - options: - - legacy - default: legacy npm_tag: description: npm tag to use for local package smoke tests required: false @@ -28,8 +21,10 @@ on: permissions: # release-shared.yml declares privileged publish jobs. They are gated by # dry_run here, but GitHub validates nested-workflow permissions at startup. + actions: read contents: write id-token: write + pull-requests: write jobs: smoke: @@ -37,7 +32,6 @@ jobs: uses: ./.github/workflows/release-shared.yml with: version: ${{ inputs.version }} - shell: ${{ inputs.shell }} npm_tag: ${{ inputs.npm_tag }} channel: beta prerelease: true diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index b3703b7f8d..7f043e20c4 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -25,18 +25,16 @@ on: description: npm package version to publish (must be unique on npm; pick the next unused version) required: true type: string - # Defaults to `true` so a stray "Run workflow" click can't accidentally - # publish — operators recovering from a stale-bytes run must consciously - # untick this when dispatching. + # Defaults to `true` so a stray "Run workflow" click can't accidentally publish; operators + # must consciously untick this when dispatching. dry_run: description: Dry run (skip actual publishing) required: false type: boolean default: true -# The release pipeline mutates external package registries, git tags, GitHub -# Releases, Homebrew, and Scoop. Keep one release per ref active at a time so -# duplicate push events cannot race the same computed version. +# The release pipeline mutates external registries, git tags, GitHub Releases, Homebrew, and +# Scoop, so keep one release per ref active at a time to avoid duplicate push events racing. concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: false @@ -82,7 +80,6 @@ jobs: outputs: should_release: ${{ steps.compute.outputs.should_release }} version: ${{ steps.compute.outputs.version }} - shell: ${{ steps.compute.outputs.shell }} npm_tag: ${{ steps.compute.outputs.npm_tag }} prerelease: ${{ steps.compute.outputs.prerelease }} brew_name: ${{ steps.compute.outputs.brew_name }} @@ -91,11 +88,9 @@ jobs: dry_run: ${{ steps.compute.outputs.dry_run }} channel: ${{ steps.compute.outputs.channel }} steps: - # semantic-release runs `git push --dry-run HEAD:` as part of - # verifyAuth even in `dry_run: true` mode, so the token must have push - # access to the protected `develop`/`main` branches. The default - # GITHUB_TOKEN doesn't, so we mint an App-installation token from the - # same App used for fast-forward + brew/scoop pushes. + # semantic-release runs `git push --dry-run HEAD:` as part of verifyAuth even in + # `dry_run: true` mode, so the token needs push access to the protected develop/main + # branches — the default GITHUB_TOKEN doesn't have it. - id: app-token if: github.event_name == 'push' uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 @@ -103,11 +98,9 @@ jobs: client-id: ${{ vars.GH_APP_CLIENT_ID }} private-key: ${{ secrets.GH_APP_PRIVATE_KEY }} permission-contents: write - # `persist-credentials: false` is required: otherwise checkout caches the - # default GITHUB_TOKEN as an `http.extraheader` in git config, and that - # Authorization header overrides the App token semantic-release puts in - # the push URL — making the dry-push identify as `github-actions[bot]` - # and get rejected by branch protection. + # `persist-credentials: false` is required: otherwise checkout caches the default + # GITHUB_TOKEN in git config, and its Authorization header overrides the App token + # semantic-release puts in the push URL, making the dry-push get rejected by branch protection. - uses: useblacksmith/checkout@6fd481652155169ed4d2f25ebaf97464f685175f # v1 with: fetch-depth: 0 @@ -152,7 +145,6 @@ jobs: fi case "$channel" in beta) - shell=legacy npm_tag=beta prerelease=true brew_name=supabase-beta @@ -160,7 +152,6 @@ jobs: publish_brew_scoop=true ;; stable) - shell=legacy npm_tag=latest prerelease=false brew_name=supabase @@ -175,7 +166,6 @@ jobs: { echo "should_release=$should_release" echo "version=$version" - echo "shell=$shell" echo "npm_tag=$npm_tag" echo "prerelease=$prerelease" echo "brew_name=$brew_name" @@ -189,19 +179,17 @@ jobs: name: Release needs: plan if: needs.plan.outputs.should_release == 'true' - # pull-requests: write is required by the nested propose-release-notes - # workflow (release-shared.yml -> propose-release-notes.yml). For nested - # reusable workflows, a called job's permissions can't exceed those granted - # to the calling job, so this must be declared here even though the propose - # job uses an App token for its actual PR creation. + # pull-requests: write is required by the nested propose-release-notes workflow + # (release-shared.yml -> propose-release-notes.yml): a called job's permissions can't exceed + # those granted to the calling job, so this must be declared here too. permissions: + actions: read contents: write id-token: write pull-requests: write uses: ./.github/workflows/release-shared.yml with: version: ${{ needs.plan.outputs.version }} - shell: ${{ needs.plan.outputs.shell }} npm_tag: ${{ needs.plan.outputs.npm_tag }} prerelease: ${{ needs.plan.outputs.prerelease == 'true' }} publish_brew_scoop: ${{ needs.plan.outputs.publish_brew_scoop == 'true' }} @@ -218,13 +206,11 @@ jobs: DF_FIREWALL_TOKEN: ${{ secrets.DF_FIREWALL_TOKEN }} LINEAR_CLI_STABLE_RELEASE_ACCESS_KEY: ${{ secrets.LINEAR_CLI_STABLE_RELEASE_ACCESS_KEY }} LINEAR_CLI_BETA_RELEASE_ACCESS_KEY: ${{ secrets.LINEAR_CLI_BETA_RELEASE_ACCESS_KEY }} + SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN: ${{ secrets.SUPABASE_E2E_CLI_LIVE_STAGING_ACCESS_TOKEN }} - # Republishes the supabase.com CLI reference, restoring the job the Go - # `tools/bumpdoc` ran before the monorepo merge (71b543255) deleted it — the - # published spec has been frozen at 2.98.2 since. Stable only: the reference - # documents the CLI users actually install, so pre-releases are skipped. - # Nothing depends on this job, so a docs-site failure cannot affect the - # already-completed release. + # Republishes the supabase.com CLI reference. Stable only: the reference documents the CLI + # users actually install, so pre-releases are skipped. Nothing depends on this job, so a + # docs-site failure cannot affect the already-completed release. docs: name: Publish reference docs needs: [plan, release] @@ -258,9 +244,8 @@ jobs: run: gh auth setup-git - name: Publish the CLI reference working-directory: apps/cli - # `shell: bash` for `-o pipefail`: the default `run:` shell would take - # the exit status of the publisher alone, so a generator that died - # part-way would look like a successful run of a truncated spec. + # `shell: bash` for `-o pipefail`: the default `run:` shell takes only the publisher's + # exit status, so a generator that died part-way would look like a successful run. shell: bash env: GH_TOKEN: ${{ steps.app-token.outputs.token }} @@ -274,12 +259,10 @@ jobs: bun scripts/generate-docs-spec.ts "$VERSION" \ | bun scripts/publish-docs-spec.ts --version "$VERSION" "${args[@]}" - # Posts to the release Slack channel once the pipeline succeeds. Listing - # `release` in `needs` without a status function in `if:` keeps the implicit - # success() gate, so this only runs when both plan and release succeeded. - # The `if:` then filters to real (non-dry-run) stable cuts; beta and dry - # runs stay silent. Nothing depends on this job, so a Slack/webhook - # failure can't affect the already-completed release. + # Listing `release` in `needs` without a status function in `if:` keeps the implicit success() + # gate, so this only runs when both plan and release succeeded; the `if:` then filters to real + # (non-dry-run) stable cuts. Nothing depends on this job, so a Slack failure can't affect the + # already-completed release. notify-slack: name: Notify Slack needs: [plan, release] @@ -295,15 +278,11 @@ jobs: secrets: SLACK_RELEASE_WEBHOOK: ${{ secrets.SLACK_RELEASE_WEBHOOK }} - # Reports a broken release on every channel. `failure()` evaluates against the - # `needs` chain, so this fires whenever `plan` or `release` (and anything in - # the reusable release-shared workflow) fails. Skipped jobs — e.g. the - # fast-forward path or a release that never started — don't count as failures, - # so this stays quiet there. Dry runs are excluded — the guard reads the - # dispatch input directly, so even a plan job that dies before recording its - # dry_run output stays quiet; an operator running one is already watching it - # live. When `plan` fails its outputs are empty, so the message falls back to - # the workflow run link as the actionable detail. + # Reports a broken release on every channel. `failure()` evaluates against the full `needs` + # chain, so this fires whenever `plan` or `release` (including the reusable release-shared + # workflow) fails. Skipped jobs — e.g. the fast-forward path, or a release that never started — + # don't count as failures. Dry runs are excluded via the dispatch input directly, so even a + # plan job that dies before recording its `dry_run` output stays quiet. notify-slack-failure: name: Notify Slack (failure) needs: [plan, release] diff --git a/.github/workflows/run-ci.yml b/.github/workflows/run-ci.yml index c3dffb9788..49abc78abd 100644 --- a/.github/workflows/run-ci.yml +++ b/.github/workflows/run-ci.yml @@ -1,11 +1,12 @@ name: run-ci -# Opt-in full develop CI for PRs that Test.yml / preview do not already cover: -# stacked PRs (base is not develop) and drafts. Ready develop PRs stay on the -# existing workflows so required check names and concurrency are unchanged. +# Opt-in full develop CI for PRs that Test.yml does not already cover: stacked +# PRs (base is not develop) and drafts. Ready develop PRs stay on the existing +# workflows so required check names and concurrency are unchanged. # # Add the `run-ci` label to start the suite; remove it to cancel in-progress # runs via this workflow's concurrency group. Other labels do not retrigger. +# Preview packages are a separate opt-in (`run-preview-packages`). on: pull_request: types: @@ -20,7 +21,6 @@ on: permissions: actions: read contents: read - pull-requests: write # Unrelated label events still start a run; give them a unique group so they # cannot cancel an in-progress suite. Removing `run-ci` stays on the main @@ -50,18 +50,3 @@ jobs: force: true secrets: DF_FIREWALL_TOKEN: ${{ secrets.DF_FIREWALL_TOKEN }} - - preview: - name: Preview packages - if: | - !startsWith(github.head_ref, 'release-notes/') && - contains(github.event.pull_request.labels.*.name, 'run-ci') && - (github.event.pull_request.draft || github.base_ref != 'develop') && - ((github.event.action != 'labeled' && - github.event.action != 'unlabeled') || - github.event.label.name == 'run-ci') - uses: ./.github/workflows/publish-preview-cli-packages.yml - with: - force: true - secrets: - DF_FIREWALL_TOKEN: ${{ secrets.DF_FIREWALL_TOKEN }} diff --git a/.github/workflows/setup-cli-smoke-test.yml b/.github/workflows/setup-cli-smoke-test.yml index 1725e4ada2..dc9aeb1ba9 100644 --- a/.github/workflows/setup-cli-smoke-test.yml +++ b/.github/workflows/setup-cli-smoke-test.yml @@ -1,13 +1,9 @@ name: setup-cli Smoke Test -# Smoke test for the `supabase/setup-cli` GitHub Action against a specific -# published CLI version. Run automatically after every beta release (called -# from release-shared.yml's `setup-cli-smoke` job) and also manually via -# workflow_dispatch when debugging setup-cli regressions or verifying that -# a previously broken environment (e.g. Alpine) is back to green. -# -# Exists primarily to catch regressions like supabase/setup-cli#427 where -# musl libc + archive layout changes broke the action on Alpine silently. +# Smoke test for the `supabase/setup-cli` GitHub Action against a specific published CLI +# version. Runs automatically after every beta release and also manually via workflow_dispatch +# when debugging setup-cli regressions. Catches regressions like supabase/setup-cli#427, where +# musl libc + archive layout changes silently broke the action on Alpine. on: workflow_call: @@ -36,9 +32,8 @@ jobs: - ubuntu-latest - macos-latest - windows-latest - # GitHub Actions doesn't allow expressions in `uses:`, so the action - # ref is selected via two `if:`-gated install steps below rather than - # interpolating `matrix.major-version` into a single `uses:` line. + # GitHub Actions doesn't allow expressions in `uses:`, so the action ref is selected via + # two `if:`-gated install steps below instead of interpolating `matrix.major-version`. major-version: - v1 - v2 @@ -49,12 +44,12 @@ jobs: steps: - name: Install Supabase CLI via setup-cli@v1 if: matrix.major-version == 'v1' - uses: supabase/setup-cli@ab058987d8d6c725971f6cf9d0b5c98467e30bd1 # v1.7.1 + uses: supabase/setup-cli@1dedf2c611547ede7232d26866dd3c56ab903bbb # v1.7.3 with: version: ${{ inputs.version }} - name: Install Supabase CLI via setup-cli@v2 if: matrix.major-version == 'v2' - uses: supabase/setup-cli@3c2f5e2ae34c34e428e8e206e2c4d21fa2d20fbf # v2.1.1 + uses: supabase/setup-cli@afb1b15109756ea5cf9d8985a359d9095235ca2b # v2.1.2 with: version: ${{ inputs.version }} - name: Verify supabase --version matches the expected version @@ -72,14 +67,10 @@ jobs: exit 1 fi - # Alpine leg of the setup-cli smoke test. Kept as a separate job because - # GitHub Actions only honours the `container:` field on Linux runners, and - # mixing container + non-container entries in a single matrix via - # `container: ${{ matrix.container }}` is fragile when the value is empty. - # - # The musl-vs-glibc + archive layout regressions tracked in - # supabase/setup-cli#427 only reproduce inside a real Alpine container, so - # this job is the actual signal the workflow was created for. + # A separate job because GitHub Actions only honours `container:` on Linux runners, and mixing + # container + non-container entries in one matrix is fragile when the value is empty. The + # musl-vs-glibc + archive layout regressions from supabase/setup-cli#427 only reproduce inside + # a real Alpine container. setup-cli-smoke-test-alpine: name: setup-cli ${{ matrix.major-version }} (alpine) strategy: @@ -90,9 +81,8 @@ jobs: - v2 runs-on: ubuntu-latest timeout-minutes: 10 - # `node:20-alpine` ships a musl-linked Node, which is required for - # JavaScript-based actions (setup-cli included) to launch inside an - # Alpine container — the runner's mounted glibc Node won't execute here. + # `node:20-alpine` ships a musl-linked Node, required for JavaScript-based actions to launch + # here — the runner's mounted glibc Node won't execute inside an Alpine container. container: image: node:20-alpine env: @@ -102,12 +92,12 @@ jobs: run: apk add --no-cache bash curl tar - name: Install Supabase CLI via setup-cli@v1 if: matrix.major-version == 'v1' - uses: supabase/setup-cli@ab058987d8d6c725971f6cf9d0b5c98467e30bd1 # v1.7.1 + uses: supabase/setup-cli@1dedf2c611547ede7232d26866dd3c56ab903bbb # v1.7.3 with: version: ${{ inputs.version }} - name: Install Supabase CLI via setup-cli@v2 if: matrix.major-version == 'v2' - uses: supabase/setup-cli@3c2f5e2ae34c34e428e8e206e2c4d21fa2d20fbf # v2.1.1 + uses: supabase/setup-cli@afb1b15109756ea5cf9d8985a359d9095235ca2b # v2.1.2 with: version: ${{ inputs.version }} - name: Verify supabase --version matches the expected version diff --git a/.github/workflows/sync-artifacts-catalog.yml b/.github/workflows/sync-artifacts-catalog.yml new file mode 100644 index 0000000000..d57b5d7b28 --- /dev/null +++ b/.github/workflows/sync-artifacts-catalog.yml @@ -0,0 +1,98 @@ +name: Sync Artifacts catalog + +# The script is the base revision. The pull request only supplies the Dockerfile +# and catalog text, and the job runs for Dependabot so that pull request cannot +# edit this workflow and then execute it with a write token. The catalog commit +# is pushed with the app token so the new head triggers CI. Dependabot's +# GITHUB_TOKEN would not. +on: + pull_request: + paths: + - apps/cli/src/shared/services/Dockerfile + +permissions: + contents: read + +concurrency: + group: sync-artifacts-catalog-${{ github.event.pull_request.number }} + cancel-in-progress: false + +jobs: + sync: + name: Pin workload catalog + if: github.actor == 'dependabot[bot]' && github.event.pull_request.head.repo.full_name == github.repository + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.event.pull_request.base.sha }} + persist-credentials: true + fetch-depth: 1 + + # Bun only. The script imports no workspace packages. The firewall token + # is empty on a Dependabot run; setup is not required to format one file. + - name: Install toolchains + uses: jdx/mise-action@c2a87611a18de5b3828c5652fe268e992400cb5c # v4.3.0 + with: + version: 2026.9.0 + + - name: Install regctl + run: | + set -euo pipefail + install -d "${RUNNER_TEMP}/regctl-bin" + curl -fsSLo "${RUNNER_TEMP}/regctl-bin/regctl" \ + https://github.com/regclient/regclient/releases/download/v0.11.5/regctl-linux-amd64 + echo "c93aa7638749f5aaac1a8e01787321889c78f0101809bb2880343478d0ba0467 ${RUNNER_TEMP}/regctl-bin/regctl" | sha256sum -c - + chmod +x "${RUNNER_TEMP}/regctl-bin/regctl" + echo "${RUNNER_TEMP}/regctl-bin" >> "$GITHUB_PATH" + "${RUNNER_TEMP}/regctl-bin/regctl" version + + - name: Generate token + id: app-token + uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 + with: + client-id: ${{ vars.GH_APP_CLIENT_ID }} + private-key: ${{ secrets.GH_APP_PRIVATE_KEY }} + permission-contents: write + + - name: Pin the catalog to the Dockerfile + env: + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + HEAD_REF: ${{ github.event.pull_request.head.ref }} + PR_NUMBER: ${{ github.event.pull_request.number }} + APP_TOKEN: ${{ steps.app-token.outputs.token }} + run: | + set -euo pipefail + git fetch --no-tags origin "pull/${PR_NUMBER}/head" + fetched="$(git rev-parse FETCH_HEAD)" + if [ "$fetched" != "$HEAD_SHA" ]; then + echo "::error ::pull/${PR_NUMBER} head is ${fetched}, expected ${HEAD_SHA}." + exit 1 + fi + dockerfile="$(mktemp)" + catalog="$(mktemp)" + original="$(mktemp)" + git show "$HEAD_SHA:apps/cli/src/shared/services/Dockerfile" > "$dockerfile" + git show "$HEAD_SHA:packages/stack/src/Artifacts.ts" > "$catalog" + cp "$catalog" "$original" + bun .github/scripts/sync-artifacts-catalog.ts "$dockerfile" "$catalog" apps/cli/src/shared/services/Dockerfile + if cmp -s "$original" "$catalog"; then + echo "Workload catalog already matches the Dockerfile." + exit 0 + fi + git checkout --detach "$HEAD_SHA" + cp "$catalog" packages/stack/src/Artifacts.ts + bun x oxfmt@0.65.0 --config .oxfmtrc.json packages/stack/src/Artifacts.ts + git add -- packages/stack/src/Artifacts.ts + if git diff --cached --quiet -- packages/stack/src/Artifacts.ts; then + echo "Workload catalog already matches the Dockerfile." + exit 0 + fi + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git commit -m "chore(stack): pin the workload catalog to the Dockerfile image tags" + git remote set-url origin "https://x-access-token:${APP_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" + git push origin "HEAD:refs/heads/${HEAD_REF}" diff --git a/.github/workflows/sync-stack-service-versions.yml b/.github/workflows/sync-stack-service-versions.yml deleted file mode 100644 index b8cdb52e6e..0000000000 --- a/.github/workflows/sync-stack-service-versions.yml +++ /dev/null @@ -1,61 +0,0 @@ -name: Sync Stack Service Versions - -on: - pull_request: - types: - - opened - - synchronize - - reopened - paths: - - apps/cli-go/pkg/config/templates/Dockerfile - -permissions: - contents: read - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number }} - cancel-in-progress: true - -jobs: - sync: - name: Sync stack service versions - runs-on: blacksmith-2vcpu-ubuntu-2404 - if: github.event.pull_request.user.login == 'dependabot[bot]' && github.repository == github.event.pull_request.head.repo.full_name - steps: - - name: Checkout - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: ${{ github.event.pull_request.head.ref }} - persist-credentials: false - - - name: Setup - uses: ./.github/actions/setup - with: - dependency-firewall-token: ${{ secrets.DF_FIREWALL_TOKEN }} - - - name: Sync stack service versions - run: pnpm sync:versions - working-directory: packages/stack - - - name: Generate token - id: app-token - uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0 - with: - client-id: ${{ vars.GH_APP_CLIENT_ID }} - private-key: ${{ secrets.GH_APP_PRIVATE_KEY }} - permission-contents: write - - - name: Commit synced stack service versions - env: - GH_APP_TOKEN: ${{ steps.app-token.outputs.token }} - run: | - if git diff --quiet -- packages/stack/src/ServiceCatalog.ts; then - echo "Stack service versions are already synced." - exit 0 - fi - - git config user.name "github-actions[bot]" - git config user.email "41898282+github-actions[bot]@users.noreply.github.com" - git add packages/stack/src/ServiceCatalog.ts - git commit -m "chore(stack): sync service version manifest" - git push "https://x-access-token:${GH_APP_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "HEAD:${GITHUB_HEAD_REF}" diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index de3dec7ff4..bc1b76d14b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -35,9 +35,8 @@ permissions: actions: read # Literal prefix: called workflows inherit github.workflow from the caller -# (`run-ci`), which would cancel the caller and the preview call. `inputs.force` -# keeps a skipped native Test run on a develop draft from cancelling the -# forced run-ci call. +# (`run-ci`), which would cancel the caller. `inputs.force` keeps a skipped +# native Test run on a develop draft from cancelling the forced run-ci call. concurrency: group: test.yml-${{ github.event.pull_request.number || github.head_ref || github.ref }}-${{ inputs.force && 'run-ci' || 'direct' }} cancel-in-progress: true @@ -122,6 +121,44 @@ jobs: - name: Run integration tests run: pnpm run test:integration --coverage.enabled + test-stack-ports: + if: | + !startsWith(github.head_ref, 'release-notes/') && + (github.event_name == 'merge_group' || + inputs.force || + github.event.pull_request.draft == false) + name: Run stack port tests (${{ matrix.os }}) + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [blacksmith-8vcpu-ubuntu-2404, macos-latest, windows-latest] + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Setup + uses: ./.github/actions/setup + with: + dependency-firewall-token: ${{ secrets.DF_FIREWALL_TOKEN }} + + - name: Run stack integration tests + working-directory: packages/stack + run: >- + pnpm test:integration + src/Ports.integration.test.ts + src/storage/DirectoryCopy.integration.test.ts + src/State.integration.test.ts + src/State.windows.integration.test.ts + src/State.process.integration.test.ts + src/HostProcess.integration.test.ts + src/Network.integration.test.ts + src/Proxy.integration.test.ts + src/HttpProxy.integration.test.ts + --passWithNoTests=false + test-summary: if: | always() && @@ -130,17 +167,21 @@ jobs: inputs.force || github.event.pull_request.draft == false) name: Run unit and integration tests - needs: [test-unit, test-integration] + needs: [test-unit, test-integration, test-stack-ports] runs-on: ubuntu-latest steps: - name: Verify unit and integration tests succeeded run: | if [ "${{ needs.test-unit.result }}" = "failure" ] || [ "${{ needs.test-unit.result }}" = "cancelled" ] || \ [ "${{ needs.test-integration.result }}" = "failure" ] || [ "${{ needs.test-integration.result }}" = "cancelled" ]; then - echo "::error ::Unit or integration tests failed: unit=${{ needs.test-unit.result }}, integration=${{ needs.test-integration.result }}" + echo "::error ::Unit or integration tests failed: unit=${{ needs.test-unit.result }}, integration=${{ needs.test-integration.result }}, stack-ports=${{ needs.test-stack-ports.result }}" + exit 1 + fi + if [ "${{ needs.test-stack-ports.result }}" = "failure" ] || [ "${{ needs.test-stack-ports.result }}" = "cancelled" ]; then + echo "::error ::Stack port tests failed: stack-ports=${{ needs.test-stack-ports.result }}" exit 1 fi - echo "Unit and integration tests reported: unit=${{ needs.test-unit.result }}, integration=${{ needs.test-integration.result }}" + echo "Unit and integration tests reported: unit=${{ needs.test-unit.result }}, integration=${{ needs.test-integration.result }}, stack-ports=${{ needs.test-stack-ports.result }}" test-e2e: if: | @@ -148,7 +189,7 @@ jobs: (github.event_name == 'merge_group' || inputs.force || github.event.pull_request.draft == false) - name: Run end-to-end tests (shard ${{ matrix.shard }}/3) + name: Run CLI end-to-end tests (shard ${{ matrix.shard }}/3) runs-on: blacksmith-8vcpu-ubuntu-2404 strategy: fail-fast: false @@ -178,24 +219,54 @@ jobs: run: go build -o supabase-go . working-directory: apps/cli-go - # The ts-legacy harness invokes `node apps/cli/dist/supabase.js` with - # `SUPABASE_CLI_BINARY_OVERRIDE` pointing at the compiled legacy binary - # in `apps/cli/dist/`. Build the CLI explicitly before invoking every + # The e2e harness invokes `node apps/cli/dist/supabase.js` with + # `SUPABASE_CLI_BINARY_OVERRIDE` pointing at the compiled binary in + # `apps/cli/dist/`. Build the CLI explicitly before invoking every # package-local e2e suite. - name: Build CLI run: pnpm exec turbo run supabase#build - name: Run end-to-end tests - run: pnpm exec turbo run test:e2e:run --only --concurrency=1 -- --shard=${{ matrix.shard }}/3 + run: pnpm exec turbo run test:e2e:run --only --concurrency=1 --filter=supabase --filter=@supabase/cli-e2e -- --shard=${{ matrix.shard }}/3 env: - CLI_HARNESS_TARGET: ts-legacy SUPABASE_GO_BINARY: ${{ github.workspace }}/apps/cli-go/supabase-go - # Summary job that gates branch protection. The matrix `test-e2e` job - # produces per-shard check names (`Run end-to-end tests (shard N/3)`), so - # this job preserves the original `Run end-to-end tests` check name that - # branch protection rules already require. It succeeds iff every shard - # succeeded (or skipped — `success()` is true for skipped jobs). + test-stack-e2e: + if: | + !startsWith(github.head_ref, 'release-notes/') && + (github.event_name == 'merge_group' || + inputs.force || + github.event.pull_request.draft == false) + name: Run stack end-to-end tests (${{ matrix.runtime }}) + runs-on: blacksmith-8vcpu-ubuntu-2404 + strategy: + fail-fast: false + matrix: + runtime: [native, docker] + steps: + - name: Checkout + uses: useblacksmith/checkout@6fd481652155169ed4d2f25ebaf97464f685175f # v1.0.0-beta + + - name: Setup + uses: ./.github/actions/setup + with: + dependency-firewall-token: ${{ secrets.DF_FIREWALL_TOKEN }} + + - name: Run stack end-to-end tests + run: >- + pnpm --filter @supabase/stack test:e2e:run + src/whole-stack.${{ matrix.runtime }}.e2e.test.ts + --passWithNoTests=false + + - name: Run public stack end-to-end tests + if: matrix.runtime == 'native' + run: >- + pnpm --filter @supabase/stack test:e2e:run + src/public.e2e.test.ts + --passWithNoTests=false + + # Summary job that gates branch protection. It preserves the original check + # name and succeeds iff every CLI shard and stack runtime succeeded. test-e2e-summary: if: | always() && @@ -204,13 +275,13 @@ jobs: inputs.force || github.event.pull_request.draft == false) name: Run end-to-end tests - needs: test-e2e + needs: [test-e2e, test-stack-e2e] runs-on: ubuntu-latest steps: - name: Verify all shards succeeded run: | - if [ "${{ needs.test-e2e.result }}" = "failure" ] || [ "${{ needs.test-e2e.result }}" = "cancelled" ]; then - echo "::error ::One or more e2e shards failed: ${{ needs.test-e2e.result }}" + if [ "${{ needs.test-e2e.result }}" = "failure" ] || [ "${{ needs.test-e2e.result }}" = "cancelled" ] || [ "${{ needs.test-stack-e2e.result }}" = "failure" ] || [ "${{ needs.test-stack-e2e.result }}" = "cancelled" ]; then + echo "::error ::One or more e2e jobs failed: cli=${{ needs.test-e2e.result }}, stack=${{ needs.test-stack-e2e.result }}" exit 1 fi - echo "All e2e shards reported: ${{ needs.test-e2e.result }}" + echo "All e2e jobs reported: cli=${{ needs.test-e2e.result }}, stack=${{ needs.test-stack-e2e.result }}" diff --git a/.github/workflows/verify-install-channels.yml b/.github/workflows/verify-install-channels.yml index cf5a6983bb..a3a94bc4fa 100644 --- a/.github/workflows/verify-install-channels.yml +++ b/.github/workflows/verify-install-channels.yml @@ -1,35 +1,14 @@ name: Verify Install Channels -# Post-publish end-to-end verification that the *published* install channels -# (Homebrew, Scoop, and the curl|bash install script) actually install the -# just-released CLI and serve artifacts whose checksums match what the channel -# manifests declare. Runs automatically after every brew/scoop publish (called -# from release-shared.yml's `verify-install-channels` job) and can also be -# dispatched manually against any already-published version when debugging an -# install regression. +# Post-publish verification that the published install channels (Homebrew, Scoop, curl|bash) +# install the just-released CLI and serve artifacts matching the channel manifests' checksums. +# Runs automatically after every brew/scoop publish, and can also be dispatched manually against +# an already-published version. # -# Exists primarily to catch regressions like CLI v2.107.0, where the Homebrew -# formula and Scoop manifest shipped sha256 checksums that did not match the -# tarballs on the GitHub Release, so `brew install` / `scoop install` failed -# for every user with "Formula reports different checksum". brew, scoop, and -# the install script all verify the declared checksum against the downloaded -# bytes before installing, so a real install reproduces that failure exactly -# instead of trusting the manifest the publish step wrote. -# -# Each leg goes beyond `supabase --version` (handled by the Bun wrapper without -# touching the sidecar) and directly checks that the `supabase-go` sidecar -# binary is present and executable in the channel's install directory, so a -# package that omits or misplaces it fails here instead of silently shipping -# a CLI whose still-Go-proxied commands (see docs/go-cli-porting-status.md) -# would fail for every user. -# -# This used to run `supabase completion bash`, since that command was -# Go-proxied. It no longer is (CLI-1965 ported shell completion to native -# TypeScript, and the Go CLI's own completion command was subsequently -# removed too), so that probe silently stopped testing the sidecar at all. -# Checking for the sidecar file directly instead of routing through some -# still-proxied command avoids repeating that mistake as more commands get -# natively ported. +# Performs a real install so a manifest/tarball checksum mismatch (as in CLI v2.107.0) reproduces +# here instead of being trusted from the publish step. Each leg also directly checks that the +# `supabase-go` sidecar binary is present and executable in the install directory, checking the +# file rather than a CLI command, since a command could later be natively ported off the sidecar. on: workflow_call: @@ -68,10 +47,8 @@ permissions: jobs: homebrew: - # macOS and Linux exercise different stanzas of the formula - # (`on_macos` vs `on_linux`), each with its own URL + sha256, so both must - # install for the tap to be considered verified. Homebrew is preinstalled - # on GitHub-hosted Ubuntu runners. + # macOS and Linux exercise different formula stanzas (`on_macos` vs `on_linux`), each with + # its own URL + sha256, so both must install for the tap to be considered verified. name: Homebrew ${{ inputs.brew_name }} (${{ matrix.runner }}) strategy: fail-fast: false @@ -84,14 +61,11 @@ jobs: env: VERSION: ${{ inputs.version }} BREW_NAME: ${{ inputs.brew_name }} - # Don't auto-update Homebrew itself before installing; `brew install - # //` taps supabase/homebrew-tap from its git HEAD - # regardless, so the freshly-pushed formula is picked up either way. + # Skips Homebrew's own auto-update; `brew install //` taps + # supabase/homebrew-tap from git HEAD regardless, so the fresh formula is picked up anyway. # - # NB: do NOT set HOMEBREW_NO_INSTALL_FROM_API here. It only affects the - # homebrew/core + cask taps (third-party taps are always read from git), - # and on Linux runners it forces a large, slow local checkout of - # homebrew/core that makes this job hang for 10+ minutes. + # Do not also set HOMEBREW_NO_INSTALL_FROM_API: it only affects homebrew/core + cask taps, + # and on Linux it forces a slow homebrew/core checkout that hangs this job for 10+ minutes. HOMEBREW_NO_AUTO_UPDATE: "1" steps: - name: Set up Homebrew on PATH @@ -205,9 +179,8 @@ jobs: scoop bucket add supabase https://github.com/supabase/scoop-bucket scoop install "supabase/$env:SCOOP_NAME" - name: Verify supabase --version - # Force bash so ${VERSION} expands the same way it does on the other - # legs — windows-latest defaults to pwsh, which treats it as an empty - # PowerShell variable (env vars are `$env:VAR`). + # Force bash so ${VERSION} expands the same way as the other legs — windows-latest + # defaults to pwsh, which treats it as an empty PowerShell variable. shell: bash run: | set -euo pipefail diff --git a/.gitignore b/.gitignore index 5a892f3aba..a1d49ff0cf 100644 --- a/.gitignore +++ b/.gitignore @@ -7,8 +7,10 @@ coverage/ !.env.example .claude/ .agents/.repos/effect-v3 +.repos/slim-services/ .worktrees/ .supabase/ +erl_crash.dump # Stray `supabase` project dir created by running the CLI at the repo root # (e.g. supabase/.temp/linked-project.json). This monorepo has no top-level # Supabase project — real fixtures live under apps/cli-e2e/fixtures/. @@ -23,6 +25,7 @@ packages/cli-*/bin/ # Turbo .turbo/ +apps/cli/.supabase/ # Transient render dir created by packages/api/scripts/generated-output-sync.unit.test.ts packages/api/.generated-output-sync-*/ diff --git a/.gitmodules b/.gitmodules index ec13d73342..f813471cd0 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,21 +1,20 @@ [submodule ".repos/effect"] path = .repos/effect - url = https://github.com/Effect-TS/effect-smol.git + url = https://github.com/Effect-TS/effect.git + branch = main [submodule ".repos/effect-patterns"] path = .repos/effect-patterns url = https://github.com/PaulJPhilp/EffectPatterns [submodule ".repos/effect-v3"] path = .repos/effect-v3 url = https://github.com/Effect-TS/effect.git + branch = v3 [submodule ".repos/lalph"] path = .repos/lalph url = https://github.com/tim-smart/lalph.git [submodule ".repos/cheffect"] path = .repos/cheffect url = https://github.com/tim-smart/cheffect.git -[submodule ".repos/process-compose"] - path = .repos/process-compose - url = https://github.com/F1bonacc1/process-compose.git [submodule ".repos/t3code"] path = .repos/t3code url = https://github.com/pingdotgg/t3code.git diff --git a/.husky/commit-msg b/.husky/commit-msg new file mode 100755 index 0000000000..2e6b87e2fc --- /dev/null +++ b/.husky/commit-msg @@ -0,0 +1 @@ +pnpm exec commitlint --edit "$1" diff --git a/.oxfmtrc.json b/.oxfmtrc.json index 0e2a3a42c0..cdddd6ab52 100644 --- a/.oxfmtrc.json +++ b/.oxfmtrc.json @@ -1,6 +1,7 @@ { "ignorePatterns": [ ".repos/", + "apps/cli/src/shared/feedback/database.types.ts", "apps/cli-go/", "apps/cli-e2e/fixtures/", "apps/docs/content/docs/commands/", diff --git a/.oxlintrc.effect.json b/.oxlintrc.effect.json index 7a1c87954f..0b7056e787 100644 --- a/.oxlintrc.effect.json +++ b/.oxlintrc.effect.json @@ -4,5 +4,74 @@ "options": { "denyWarnings": true }, - "ignorePatterns": [] + "ignorePatterns": [ + "**", + "!packages/stack/**", + "!apps/cli/src/commands/backups/**", + "!apps/cli/src/commands/bootstrap/**", + "!apps/cli/src/commands/branches/**", + "!apps/cli/src/commands/completion/**", + "!apps/cli/src/commands/config/**", + "!apps/cli/src/commands/db/**", + "!apps/cli/src/commands/domains/**", + "!apps/cli/src/commands/encryption/**", + "!apps/cli/src/commands/experimental/**", + "!apps/cli/src/commands/feedback/**", + "!apps/cli/src/commands/functions/**", + "!apps/cli/src/commands/gen/**", + "!apps/cli/src/commands/init/**", + "!apps/cli/src/commands/inspect/**", + "!apps/cli/src/commands/issue/**", + "!apps/cli/src/commands/link/**", + "!apps/cli/src/commands/login/**", + "!apps/cli/src/commands/logout/**", + "!apps/cli/src/commands/migration/**", + "!apps/cli/src/commands/network-bans/**", + "!apps/cli/src/commands/network-restrictions/**", + "!apps/cli/src/commands/notebooks/**", + "!apps/cli/src/commands/orgs/**", + "!apps/cli/src/commands/postgres-config/**", + "!apps/cli/src/commands/projects/**", + "!apps/cli/src/commands/pull/**", + "!apps/cli/src/commands/secrets/**", + "!apps/cli/src/commands/seed/**", + "!apps/cli/src/commands/services/**", + "!apps/cli/src/commands/snippets/**", + "!apps/cli/src/commands/ssl-enforcement/**", + "!apps/cli/src/commands/sso/**", + "!apps/cli/src/commands/status/**", + "!apps/cli/src/commands/stop/**", + "!apps/cli/src/commands/storage/**", + "!apps/cli/src/commands/telemetry/**", + "!apps/cli/src/commands/test/**", + "!apps/cli/src/commands/unlink/**", + "!apps/cli/src/commands/vanity-subdomains/**", + "!apps/cli/src/commands/whoami/**", + "!apps/cli/src/shared/compute/**", + "!apps/cli/src/shared/functions/functions-docker.ts", + "!apps/cli/src/shared/functions/functions-docker.unit.test.ts", + "!apps/cli/src/shared/functions/serve.ts", + "!apps/cli/src/shared/functions/serve.unit.test.ts", + "!apps/cli/src/shared/runtime/file-watcher.service.ts", + "!apps/cli/src/commands/start/**", + // Last match wins across the whole list: keep bare re-exclusions after every + // `!` entry that would otherwise re-include them. + "apps/cli/src/shared/compute/stacks/**", + "!apps/cli/tests/helpers/branches-live.ts", + "!apps/cli/tests/helpers/compute.ts", + "!apps/cli/tests/helpers/migration-live.ts", + "!apps/cli/tests/helpers/notebooks.ts", + "!apps/cli/tests/helpers/postgres-config-live.ts", + "!apps/cli/tests/helpers/secrets-live.ts", + "!apps/cli/tests/helpers/storage-live.ts", + "!apps/cli/src/command-internal/experimental-feature.ts", + "!apps/cli/src/command-internal/postgres-client.run.ts", + "!apps/cli/src/command-internal/stack-api.ts", + "!apps/cli/src/command-internal/stack-backend.ts", + "!apps/cli/src/command-internal/stack-catalog-setup.ts", + "!apps/cli/src/command-internal/stack-config.ts", + "!apps/cli/src/command-internal/stack-local-database.ts", + "!apps/cli/src/command-internal/stack-shadow.ts", + "!apps/cli/src/command-internal/stack-storage.ts" + ] } diff --git a/.oxlintrc.json b/.oxlintrc.json index 4d587a2f9a..8c10b88c04 100644 --- a/.oxlintrc.json +++ b/.oxlintrc.json @@ -7,9 +7,9 @@ "ignorePatterns": [ ".repos", "apps/cli-go", + // Generated by `supabase gen types` (see `gen:feedback-types` in apps/cli). + "apps/cli/src/shared/feedback/database.types.ts", "apps/cli-e2e/fixtures", - "packages/stack", - "packages/process-compose", "**/testdata", "**/dist", "**/coverage", @@ -17,11 +17,10 @@ "**/.source" ], "rules": { - // Spreading an Effect Data.Class instance into its constructor is the - // idiomatic copy-with-update pattern (`new State({ ...state, ... })`). + // `new State({ ...state, ... })` is the idiomatic Data.Class copy-with-update pattern. "typescript/no-misused-spread": "off", - // Effect combinators are passed as references by design, e.g. - // `.pipe(semaphore.withPermit)`; they do not rely on `this`. + // Effect combinators like `.pipe(semaphore.withPermit)` are passed as references by + // design and don't rely on `this`. "typescript/unbound-method": "off", // Too strict for CLI string building; `no-base-to-string` still guards // against `[object Object]` interpolation bugs. @@ -29,7 +28,7 @@ }, "overrides": [ { - // The legacy CLI is the stable, shipped shell; a lint-driven rewrite of + // The CLI is the stable, shipped tool; a lint-driven rewrite of // its existing string-building code isn't worth the churn. "files": ["apps/cli/src/**"], "rules": { @@ -37,13 +36,9 @@ } }, { - // `.github/scripts` is the only `bun:test` consumer in the repo (every - // package test suite uses vitest). `@types/bun`'s test matcher types - // reuse the same sync `Matchers` interface for `expect(x).rejects`, - // so `.rejects.toThrow(...)` types as returning `void` even though it - // must be awaited at runtime — a `@types/bun` typing gap, not a real - // `await`-of-non-Promise bug. Verified: `bun test` and `tsc --noEmit` - // both pass; removing the `await` would make the assertion racy. + // `.github/scripts` is the only `bun:test` consumer; `@types/bun`'s shared `Matchers` + // types `.rejects.toThrow(...)` as returning `void`, even though it must be awaited at + // runtime. Removing the `await` would make the assertion racy. "files": [".github/scripts/**"], "rules": { "typescript/await-thenable": "off" diff --git a/.repos/cheffect b/.repos/cheffect index f4c923c96b..0d3cce8926 160000 --- a/.repos/cheffect +++ b/.repos/cheffect @@ -1 +1 @@ -Subproject commit f4c923c96b7153671b4a2c9474c1e7e03ab8f36c +Subproject commit 0d3cce8926271b0e99de3601ad45b4645c2e1bfd diff --git a/.repos/effect b/.repos/effect index 455ecc9ac7..90695b0d5b 160000 --- a/.repos/effect +++ b/.repos/effect @@ -1 +1 @@ -Subproject commit 455ecc9ac70f8495bd829e42a92d416d0b3fc730 +Subproject commit 90695b0d5bee3aa33e67cbea1aa2e6298816a64f diff --git a/.repos/effect-v3 b/.repos/effect-v3 index e71ba68273..1af4232fea 160000 --- a/.repos/effect-v3 +++ b/.repos/effect-v3 @@ -1 +1 @@ -Subproject commit e71ba68273026a1a2c1ace7218bdb206b0d3386d +Subproject commit 1af4232fea7bc613e1dc68db9bec7b1f596d9e68 diff --git a/.repos/lalph b/.repos/lalph index 0fddf20a7d..ea5f0044a8 160000 --- a/.repos/lalph +++ b/.repos/lalph @@ -1 +1 @@ -Subproject commit 0fddf20a7d70391bb583fb0abadc4223b618ec4d +Subproject commit ea5f0044a8ef439597f87308067bc7bd289cac42 diff --git a/.repos/opencode b/.repos/opencode index 6e4db5666a..e03db9bc69 160000 --- a/.repos/opencode +++ b/.repos/opencode @@ -1 +1 @@ -Subproject commit 6e4db5666ae33ebadf3b8ca077d6b1b149d0b0c3 +Subproject commit e03db9bc6908f75c9334d8aa997deeaac81c0298 diff --git a/.repos/process-compose b/.repos/process-compose deleted file mode 160000 index a4038d6698..0000000000 --- a/.repos/process-compose +++ /dev/null @@ -1 +0,0 @@ -Subproject commit a4038d669818c35fc68fc7fc240b39e371ce0e7a diff --git a/.repos/t3code b/.repos/t3code index d1e85c4e8f..6ecc15fa25 160000 --- a/.repos/t3code +++ b/.repos/t3code @@ -1 +1 @@ -Subproject commit d1e85c4e8fdef82fbaded9539532b754080419e0 +Subproject commit 6ecc15fa25c808922a58f07377add25b55d9ac90 diff --git a/AGENTS.md b/AGENTS.md index 9e313dc366..7cdec80512 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,395 +1,161 @@ # Supabase -Bun monorepo with workspaces under `apps/` and `packages/`. +Bun monorepo with workspaces under `apps/` and `packages/`. `pnpm` is the package manager; use +`pnpm