Skip to content

docs: add a no-credential quickstart for landing a change - #591

Merged
behinddwalls merged 1 commit into
mainfrom
demo-docs
Aug 14, 2026
Merged

docs: add a no-credential quickstart for landing a change#591
behinddwalls merged 1 commit into
mainfrom
demo-docs

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

The on-ramp jumped straight from "Ping returns" to "get a scratch repository and a GitHub token". Nothing in between showed the queue doing the thing it exists to do, even though the local stack has been able to land a change with no credentials all along — every integration at the edges is faked, so a request walks the whole pipeline in seconds and settles at landed.

The Ping step it stopped at did not work either. The gateway registers reflection, but its descriptor references api/base/change/proto/change.proto while the generated change.pb.go registers that file as plain change.proto, so reflection cannot resolve the service and grpcurl refuses every method on it. The grpcurl invocation in both README.md and DEVELOPMENT.md therefore failed for anyone who ran it.

service/submitqueue/demo/provider/local/ looked like the natural home for this, but it is not reachable by hand: make local-provider-start PROVIDER=local layers docker-compose.provider.yml, which hard-requires GITHUB_TOKEN, and the overlay it actually wants needs a bare repository the E2E harness provisions. Making that runnable is a Makefile change rather than a docs change, so this keeps to what already runs.

What?

A new doc/howto/QUICKSTART.md: start the stack, discover the gateway's port, land a change, follow it to landed, then make one fail on demand with a ?sq-fake=build-fail marker and watch a good change land behind it. It covers the two things that reject a request outright — a 40-hex commit SHA and a percent-encoded refs/… ref — and is explicit that faking the edges is what makes the run free and what stops landed from proving a commit reached a branch.

README.md's Quick Start becomes that run in six commands, replacing the broken grpcurl line, and DEVELOPMENT.md's "Try It Locally" matches. Both link on to make e2e-git-test and PROVIDER-E2E.md as the tiers that add a real merge and a real provider.

PROVIDER-E2E.md carried a stale claim that the finer statuses "are never published", so a request "sits on batched for the whole of its active life". The request log disagrees: accepted → started → validating → validated → batched → speculating → speculated → landing → landed, with building and built recorded as events. That paragraph, the sentence repeating it further down, and the sample table above it are corrected.

Two defects surfaced while verifying this and are documented but not fixed here, since both are code changes:

  • the reflection descriptor mismatch above, which makes grpcurl unusable against the gateway;
  • both MySQL services mount anonymous volumes, so every stop/start cycle orphans a pair and comes back to an empty database — make local-stop reports "Data volumes preserved", which is true of the volumes and not of the data. It is also why the volumes accumulate until Docker runs out of disk.

Test Plan

Every command in the quickstart was run against a live stack, and the documented output is what it printed.

  • make local-submitqueue-start, then the documented make land / make land-status — reached landed
  • ✅ the same URI with ?sq-fake=build-fail — terminal error
  • ✅ a failing change and a good one in flight together — the first went error, the second still reached landed
  • ✅ the trail in both the request_log table and make land-watch, which is what the corrected PROVIDER-E2E.md paragraph now states
  • ✅ re-ran the whole documented sequence from a torn-down stack, confirming the port really is different each time and that test-queue/1 is reproducible

Not verified: the wrapping of the sample table in PROVIDER-E2E.md, which comes from make demo-pr and needs a token. Only the status names in it were corrected; it remains illustrative.

## Summary

### Why?

The on-ramp jumped straight from "Ping returns" to "get a scratch repository and a GitHub token". Nothing in between showed the queue doing the thing it exists to do, even though the local stack has been able to land a change with no credentials all along — every integration at the edges is faked, so a request walks the whole pipeline in seconds and settles at `landed`.

The Ping step it stopped at did not work either. The gateway registers reflection, but its descriptor references `api/base/change/proto/change.proto` while the generated `change.pb.go` registers that file as plain `change.proto`, so reflection cannot resolve the service and `grpcurl` refuses every method on it. The `grpcurl` invocation in both README.md and DEVELOPMENT.md therefore failed for anyone who ran it.

`service/submitqueue/demo/provider/local/` looked like the natural home for this, but it is not reachable by hand: `make local-provider-start PROVIDER=local` layers `docker-compose.provider.yml`, which hard-requires `GITHUB_TOKEN`, and the overlay it actually wants needs a bare repository the E2E harness provisions. Making that runnable is a Makefile change rather than a docs change, so this keeps to what already runs.

### What?

A new [doc/howto/QUICKSTART.md](doc/howto/QUICKSTART.md): start the stack, discover the gateway's port, land a change, follow it to `landed`, then make one fail on demand with a `?sq-fake=build-fail` marker and watch a good change land behind it. It covers the two things that reject a request outright — a 40-hex commit SHA and a percent-encoded `refs/…` ref — and is explicit that faking the edges is what makes the run free and what stops `landed` from proving a commit reached a branch.

README.md's Quick Start becomes that run in six commands, replacing the broken `grpcurl` line, and DEVELOPMENT.md's "Try It Locally" matches. Both link on to `make e2e-git-test` and PROVIDER-E2E.md as the tiers that add a real merge and a real provider.

PROVIDER-E2E.md carried a stale claim that the finer statuses "are never published", so a request "sits on `batched` for the whole of its active life". The request log disagrees: `accepted → started → validating → validated → batched → speculating → speculated → landing → landed`, with `building` and `built` recorded as events. That paragraph, the sentence repeating it further down, and the sample table above it are corrected.

Two defects surfaced while verifying this and are documented but **not** fixed here, since both are code changes:

- the reflection descriptor mismatch above, which makes `grpcurl` unusable against the gateway;
- both MySQL services mount *anonymous* volumes, so every stop/start cycle orphans a pair and comes back to an empty database — `make local-stop` reports "Data volumes preserved", which is true of the volumes and not of the data. It is also why the volumes accumulate until Docker runs out of disk.

## Test Plan

Every command in the quickstart was run against a live stack, and the documented output is what it printed.

- ✅ `make local-submitqueue-start`, then the documented `make land` / `make land-status` — reached `landed`
- ✅ the same URI with `?sq-fake=build-fail` — terminal `error`
- ✅ a failing change and a good one in flight together — the first went `error`, the second still reached `landed`
- ✅ the trail in both the `request_log` table and `make land-watch`, which is what the corrected PROVIDER-E2E.md paragraph now states
- ✅ re-ran the whole documented sequence from a torn-down stack, confirming the port really is different each time and that `test-queue/1` is reproducible

Not verified: the wrapping of the sample table in PROVIDER-E2E.md, which comes from `make demo-pr` and needs a token. Only the status names in it were corrected; it remains illustrative.
@behinddwalls
behinddwalls marked this pull request as ready for review August 14, 2026 17:34
@behinddwalls
behinddwalls requested review from a team and sbalabanov as code owners August 14, 2026 17:34
@behinddwalls
behinddwalls added this pull request to the merge queue Aug 14, 2026
Merged via the queue into main with commit 734e13f Aug 14, 2026
27 checks passed
@behinddwalls
behinddwalls deployed to stack-rebase August 14, 2026 19:33 — with GitHub Actions Active
@behinddwalls
behinddwalls deleted the demo-docs branch August 14, 2026 19:33
behinddwalls added a commit that referenced this pull request Aug 14, 2026
## Summary

### Why?

The repository had two divergent local paths, and the quickstart documented the wrong one.

`make local-submitqueue-start` brings up an all-fake stack — fake change provider, fake CI, **noop merger** — and `doc/howto/QUICKSTART.md` (#591) described that. It reaches `landed` in seconds with no credentials, but nothing is ever pushed, so it proves the pipeline's choreography and nothing about merging. `make local-provider-start` runs the real thing, but was GitHub-only: it hardcoded the token-requiring overlay, so the credential-free provider configuration that already existed could not be reached by hand at all. `PROVIDER=local` failed at `docker compose up` on a missing `GITHUB_TOKEN`, and would have merged nothing even past that, since the overlay it needs bind-mounts a repository the other one has never heard of.

The two are not variants of one thing. Faking every edge and merging into a real repository answer different questions, and calling both "local" is what made them hard to keep apart.

### What?

`PROVIDER` now names three modes, and is the only thing that changes between them:

| `PROVIDER` | A change is | Landing it | Needs |
|---|---|---|---|
| `fake` (default) | a URI, and nothing else | reports success without touching a repository | nothing |
| `git` | a branch in a bare repository on disk | a real fetch, cherry-pick and push | nothing |
| `github` | a real pull request | a real push to a real repository | a repository and a token |

They are the tiers `PROVIDER-E2E.md` already named, so the documentation collapses into one ladder where every rung takes the same commands.

**Provider directories.** `demo/provider/local/` becomes `demo/provider/git/` — both old names were "local" — and gains a `demo-queue` entry so the default `QUEUE` works there. It needs its own `checkoutPath`: Runway refuses two queues that share a checkout with differing merger configuration, and `demo-queue` squash-rebases where `e2e-git-queue` rebases. A new `demo/provider/fake/` spells out the all-fake mode rather than leaving it implied by absent configuration. `e2e-git-queue` is untouched, and `make e2e-git-test` is the gate that proves it.

**Overlay per mode.** `PROVIDER_COMPOSE_FILE` becomes a map, since which overlay a mode needs is not something its two config files can express — `github` requires a credential, `git` requires the sandbox mounted, `fake` requires neither. A new `docker-compose.fake.yml` covers the third.

**`make demo-pr` works without GitHub.** `createOne` was the only function that touched the provider; it now describes a change and hands it to a `changeSource`. Three implement it: `fakeSource` (no I/O at all — mints a reproducible `git://` URI), `gitSource` (pushes real branches with the pinned `@git//:git`, serializing its commands because one working tree cannot take concurrent checkouts), and `githubSource` (today's REST client, extracted unchanged). `GITHUB_TOKEN` is now read only when it is actually needed.

**`tool/gitsandbox`** provisions the bare repository `PROVIDER=git` merges into, idempotently, so a restart keeps whatever landed. `platform/gitexec` locates git and strips the ambient environment, so a developer's hooks or signing key cannot fail a demo.

Two fixes fell out of getting `PROVIDER=git` to work at all:

- Runway's checkout directory moves to a named volume unless a path is given. It is where git clones, cherry-picks and commits, and on macOS a freshly written loose object read back over a bind mount can report `loose object … is corrupt` — which failed the first land against every new stack. The E2E still passes a path and still gets a bind mount.
- The Runway image now creates `/var/runway/checkouts`, so a named volume mounted there starts with a mode the service can write. Without it the service fails at boot with `mkdir: permission denied` whenever it runs as a non-root user.

## Test Plan

Every command in the rewritten quickstart was run by hand against a live stack, and the output quoted in it is what it printed.

- ✅ `make local-provider-start` (defaults to `fake`) then `make demo-pr` — three changes, no repository, no token, all `landed`
- ✅ `PROVIDER=git make local-provider-start` from a clean slate, then `PROVIDER=git make demo-pr` — real branches and multi-file commits, verified with `git -C /tmp/sq-sandbox/sandbox.git log --oneline main`
- ✅ `PROVIDER=git make demo-pr STACKED=true` — three changes, exactly one more entry in the target's reflog, so the stack landed atomically
- ✅ first land against a brand-new stack, repeatedly, which is the case the named volume fixes
- ✅ `make land` by hand on both `fake` and `git`, confirming the fake run lands nothing and the git run puts a commit on the branch
- ✅ `make e2e-git-test` after the rename and again after the compose change
- ✅ `make test` (104 targets), `make gazelle`, `make fmt`

`PROVIDER=github` is unchanged in behaviour but needs a token, so it has not been re-run here; the GitHub source is the previous code path moved behind the interface.

## Note for reviewers

`make local-provider-start` with no arguments meant GitHub before this change and means `fake` after it.
behinddwalls added a commit that referenced this pull request Aug 14, 2026
## Summary

### Why?

The repository had two divergent local paths, and the quickstart documented the wrong one.

`make local-submitqueue-start` brings up an all-fake stack — fake change provider, fake CI, **noop merger** — and `doc/howto/QUICKSTART.md` (#591) described that. It reaches `landed` in seconds with no credentials, but nothing is ever pushed, so it proves the pipeline's choreography and nothing about merging. `make local-provider-start` runs the real thing, but was GitHub-only: it hardcoded the token-requiring overlay, so the credential-free provider configuration that already existed could not be reached by hand at all. `PROVIDER=local` failed at `docker compose up` on a missing `GITHUB_TOKEN`, and would have merged nothing even past that, since the overlay it needs bind-mounts a repository the other one has never heard of.

The two are not variants of one thing. Faking every edge and merging into a real repository answer different questions, and calling both "local" is what made them hard to keep apart.

### What?

`PROVIDER` now names three modes, and is the only thing that changes between them:

| `PROVIDER` | A change is | Landing it | Needs |
|---|---|---|---|
| `fake` (default) | a URI, and nothing else | reports success without touching a repository | nothing |
| `git` | a branch in a bare repository on disk | a real fetch, cherry-pick and push | nothing |
| `github` | a real pull request | a real push to a real repository | a repository and a token |

They are the tiers `PROVIDER-E2E.md` already named, so the documentation collapses into one ladder where every rung takes the same commands.

**Provider directories.** `demo/provider/local/` becomes `demo/provider/git/` — both old names were "local" — and gains a `demo-queue` entry so the default `QUEUE` works there. It needs its own `checkoutPath`: Runway refuses two queues that share a checkout with differing merger configuration, and `demo-queue` squash-rebases where `e2e-git-queue` rebases. A new `demo/provider/fake/` spells out the all-fake mode rather than leaving it implied by absent configuration. `e2e-git-queue` is untouched, and `make e2e-git-test` is the gate that proves it.

**Overlay per mode.** `PROVIDER_COMPOSE_FILE` becomes a map, since which overlay a mode needs is not something its two config files can express — `github` requires a credential, `git` requires the sandbox mounted, `fake` requires neither. A new `docker-compose.fake.yml` covers the third.

**`make demo-pr` works without GitHub.** `createOne` was the only function that touched the provider; it now describes a change and hands it to a `changeSource`. Three implement it: `fakeSource` (no I/O at all — mints a reproducible `git://` URI), `gitSource` (pushes real branches with the pinned `@git//:git`, serializing its commands because one working tree cannot take concurrent checkouts), and `githubSource` (today's REST client, extracted unchanged). `GITHUB_TOKEN` is now read only when it is actually needed.

**`tool/gitsandbox`** provisions the bare repository `PROVIDER=git` merges into, idempotently, so a restart keeps whatever landed. `platform/gitexec` locates git and strips the ambient environment, so a developer's hooks or signing key cannot fail a demo.

Two fixes fell out of getting `PROVIDER=git` to work at all:

- Runway's checkout directory moves to a named volume unless a path is given. It is where git clones, cherry-picks and commits, and on macOS a freshly written loose object read back over a bind mount can report `loose object … is corrupt` — which failed the first land against every new stack. The E2E still passes a path and still gets a bind mount.
- The Runway image now creates `/var/runway/checkouts`, so a named volume mounted there starts with a mode the service can write. Without it the service fails at boot with `mkdir: permission denied` whenever it runs as a non-root user.

## Test Plan

Every command in the rewritten quickstart was run by hand against a live stack, and the output quoted in it is what it printed.

- ✅ `make local-provider-start` (defaults to `fake`) then `make demo-pr` — three changes, no repository, no token, all `landed`
- ✅ `PROVIDER=git make local-provider-start` from a clean slate, then `PROVIDER=git make demo-pr` — real branches and multi-file commits, verified with `git -C /tmp/sq-sandbox/sandbox.git log --oneline main`
- ✅ `PROVIDER=git make demo-pr STACKED=true` — three changes, exactly one more entry in the target's reflog, so the stack landed atomically
- ✅ first land against a brand-new stack, repeatedly, which is the case the named volume fixes
- ✅ `make land` by hand on both `fake` and `git`, confirming the fake run lands nothing and the git run puts a commit on the branch
- ✅ `make e2e-git-test` after the rename and again after the compose change
- ✅ `make test` (104 targets), `make gazelle`, `make fmt`

`PROVIDER=github` is unchanged in behaviour but needs a token, so it has not been re-run here; the GitHub source is the previous code path moved behind the interface.

## Note for reviewers

`make local-provider-start` with no arguments meant GitHub before this change and means `fake` after it.
behinddwalls added a commit that referenced this pull request Aug 14, 2026
## Summary

### Why?

The repository had two divergent local paths, and the quickstart documented the wrong one.

`make local-submitqueue-start` brought up an all-fake stack — fake change provider, fake CI, **noop merger** — and `doc/howto/QUICKSTART.md` (#591) described that. It reached `landed` in seconds with no credentials, but nothing was ever pushed, so it proved the pipeline's choreography and nothing about merging. `make local-provider-start` ran the real thing, but was GitHub-only: it hardcoded the token-requiring overlay, so the credential-free provider configuration that already existed could not be reached by hand at all. `PROVIDER=local` failed at `docker compose up` on a missing `GITHUB_TOKEN`, and would have merged nothing past that, since the overlay it needs bind-mounts a repository the other one has never heard of.

Those two are not variants of one thing. Faking every edge and merging into a real repository answer different questions, and calling both "local" is what made them hard to keep apart.

Three further confusions came out of reading the result back:

- **Two stack families did the same thing.** Once `fake` was a first-class provider, `local-submitqueue-start` and `local-provider-start` were the same run under *different compose project names* — which had already produced a wrong instruction, since `make local-submitqueue-ps` inspects project `submitqueue` and reports "not running" for a stack started as `submitqueue-provider`.
- **One document mixed two audiences.** `PROVIDER-E2E.md` opened with a tier table pairing bazel test targets with hand-run demo commands, as though `make e2e-test` and "open a pull request and watch it land" were rungs of one ladder. Automated suites belong to testing, which `TESTING.md` already documents.
- **`demo-pr` was named for something two of the three modes do not have.** A fake change is a URI and a git change is a branch; neither is a pull request.

### What?

`PROVIDER` now names three modes, and is the only thing that changes between them:

| `PROVIDER` | A change is | Landing it | Needs |
|---|---|---|---|
| `fake` (default) | a URI, and nothing else | reports success without touching a repository | nothing |
| `git` | a branch in a bare repository on disk | a real fetch, cherry-pick and push | nothing |
| `github` | a real pull request | a real push to a real repository | a repository and a token |

**Provider directories.** `demo/provider/local/` becomes `demo/provider/git/` — both old names were "local" — and gains a `demo-queue` entry so the default `QUEUE` works there. It needs its own `checkoutPath`: Runway refuses two queues that share a checkout with differing merger configuration, and `demo-queue` squash-rebases where `e2e-git-queue` rebases. A new `demo/provider/fake/` spells out the all-fake mode rather than leaving it implied by absent configuration. `e2e-git-queue` is untouched, and `make e2e-git-test` is the gate that proves it.

**One stack target.** `local-submitqueue-start` takes `PROVIDER` and layers that mode's overlay; `local-provider-start`, `-stop` and `-clean` are gone. One compose project again, so `ps`, `logs`, `stop` and `clean` all describe whatever is running. Which overlay a mode needs is a Makefile map, since it is not something the two config files can express — `github` requires a credential, `git` requires the sandbox mounted, `fake` neither — and a new `docker-compose.fake.yml` covers the third.

**One demo document.** `PROVIDER-E2E.md` is merged into `QUICKSTART.md`, which walks all three rungs in order and carries what only that document had: the token permission table, the GitHub configuration, why a landed pull request shows as *Merged*, wiring real GitHub Actions, and the GitHub-specific failure modes. Its provider-neutral operational material — `land-list` / `land-watch`, `SQ_TOKEN`, service logs, `QUEUE_LOG_LEVEL` — moves with it, into the document people read first. No test target is mentioned in any of it.

**`make demo-requests`** (was `demo-pr`, in `service/submitqueue/demo/requests`) works without GitHub. `createOne` was the only function that touched the provider; it now describes a change and hands it to a `changeSource`. Three implement it: `fakeSource` (no I/O at all — mints a reproducible `git://` URI), `gitSource` (pushes real branches with the pinned `@git//:git`, serializing its commands because one working tree cannot take concurrent checkouts), and `githubSource` (the REST client, extracted unchanged). `GITHUB_TOKEN` is read only when it is needed.

**Folder overlap, made real.** A change writes all its files into one of three folders (`demo/alpha`, `demo/beta`, `demo/gamma`) chosen from the run tag, and states the paths it touched on its change URI (`sq-files=`, beside the existing `sq-fake=`) for the fake change provider to report back. All three profiles then use `pathoverlap` by directory, so a run shows both halves of the queue's behaviour: changes sharing a folder are batched in order and speculate on each other, changes in different folders go out beside each other. The previous layout could not do this — it hashed each file's own name into two levels of 256 buckets, so a twelve-file run had roughly a one-in-a-thousand chance of any two files sharing a directory. The change is the bucket count and the granularity, not the naming.

**`tool/gitsandbox`** provisions the bare repository `PROVIDER=git` merges into, idempotently, so a restart keeps whatever landed. **`platform/gitexec`** locates git and strips the ambient environment, so a developer's hooks or signing key cannot fail a demo.

Three fixes fell out of getting this to work at all:

- Runway's checkout directory moves to a named volume unless a path is given. It is where git clones, cherry-picks and commits, and on macOS a freshly written loose object read back over a bind mount can report `loose object … is corrupt` — which failed the first land against every new stack. The E2E still passes a path and still gets a bind mount.
- The Runway image now creates `/var/runway/checkouts`, so a named volume mounted there starts with a mode the service can write. Without it the service failed at boot with `mkdir: permission denied` whenever it ran as a non-root user.
- Change URIs are storage keys capped at 255 bytes, so the file marker names one path per directory rather than every file — for a directory-keyed analyzer that is the same set of keys — and drops paths that would not fit rather than truncating one into a different directory.

## Test Plan

Every command in the rewritten quickstart was run by hand against a live stack, and the output quoted in it is what it printed.

- ✅ `make local-submitqueue-start` (defaults to `fake`) then `make demo-requests COUNT=6` — all six landed
- ✅ inspected `batch_dependent` for that run: the four `gamma` changes form one dependency chain, the two `beta` changes form their own, and **no dependency crosses folders**. Batches from before this change show the old `all` behaviour, every batch depending on every later one
- ✅ `PROVIDER=git make local-submitqueue-start` from a clean slate, then `PROVIDER=git make demo-requests` — real branches and multi-file commits, verified with `git -C /tmp/sq-sandbox/sandbox.git log --oneline main`
- ✅ `PROVIDER=git make demo-requests STACKED=true` — three changes, exactly one more entry in the target's reflog, so the stack landed atomically
- ✅ first land against a brand-new stack, repeatedly, which is the case the named volume fixes
- ✅ the 255-byte limit is a real one this hit first: an eight-file change was rejected with `change URI exceeds 255 bytes` before the marker was budgeted. Covered now by a unit test that builds a URI from `changeFilePath` output and asserts both the length and that it still parses
- ✅ `make test` (105 targets), `make lint`, `make check-gazelle`, `make check-tidy`
- ✅ `make e2e-test` and `make e2e-git-test`, after the rename and again after the compose change

`PROVIDER=github` is unchanged in behaviour but needs a token, so it has not been re-run; the GitHub source is the previous code path moved behind the interface.

## Note for reviewers

`make local-provider-start` is gone — use `make local-submitqueue-start`, which with no arguments now means `fake` where the old provider target meant GitHub.
behinddwalls added a commit that referenced this pull request Aug 14, 2026
## Summary

### Why?

The repository had two divergent local paths, and the quickstart documented the wrong one.

`make local-submitqueue-start` brought up an all-fake stack — fake change provider, fake CI, **noop merger** — and `doc/howto/QUICKSTART.md` (#591) described that. It reached `landed` in seconds with no credentials, but nothing was ever pushed, so it proved the pipeline's choreography and nothing about merging. `make local-provider-start` ran the real thing, but was GitHub-only: it hardcoded the token-requiring overlay, so the credential-free provider configuration that already existed could not be reached by hand at all. `PROVIDER=local` failed at `docker compose up` on a missing `GITHUB_TOKEN`, and would have merged nothing past that, since the overlay it needs bind-mounts a repository the other one has never heard of.

Those two are not variants of one thing. Faking every edge and merging into a real repository answer different questions, and calling both "local" is what made them hard to keep apart.

Three further confusions came out of reading the result back:

- **Two stack families did the same thing.** Once `fake` was a first-class provider, `local-submitqueue-start` and `local-provider-start` were the same run under *different compose project names* — which had already produced a wrong instruction, since `make local-submitqueue-ps` inspects project `submitqueue` and reports "not running" for a stack started as `submitqueue-provider`.
- **One document mixed two audiences.** `PROVIDER-E2E.md` opened with a tier table pairing bazel test targets with hand-run demo commands, as though `make e2e-test` and "open a pull request and watch it land" were rungs of one ladder. Automated suites belong to testing, which `TESTING.md` already documents.
- **`demo-pr` was named for something two of the three modes do not have.** A fake change is a URI and a git change is a branch; neither is a pull request.

### What?

`PROVIDER` now names three modes, and is the only thing that changes between them:

| `PROVIDER` | A change is | Landing it | Needs |
|---|---|---|---|
| `fake` (default) | a URI, and nothing else | reports success without touching a repository | nothing |
| `git` | a branch in a bare repository on disk | a real fetch, cherry-pick and push | nothing |
| `github` | a real pull request | a real push to a real repository | a repository and a token |

**Provider directories.** `demo/provider/local/` becomes `demo/provider/git/` — both old names were "local" — and gains a `demo-queue` entry so the default `QUEUE` works there. It needs its own `checkoutPath`: Runway refuses two queues that share a checkout with differing merger configuration, and `demo-queue` squash-rebases where `e2e-git-queue` rebases. A new `demo/provider/fake/` spells out the all-fake mode rather than leaving it implied by absent configuration. `e2e-git-queue` is untouched, and `make e2e-git-test` is the gate that proves it.

**One stack target.** `local-submitqueue-start` takes `PROVIDER` and layers that mode's overlay; `local-provider-start`, `-stop` and `-clean` are gone. One compose project again, so `ps`, `logs`, `stop` and `clean` all describe whatever is running. Which overlay a mode needs is a Makefile map, since it is not something the two config files can express — `github` requires a credential, `git` requires the sandbox mounted, `fake` neither — and a new `docker-compose.fake.yml` covers the third.

**One demo document.** `PROVIDER-E2E.md` is merged into `QUICKSTART.md`, which walks all three rungs in order and carries what only that document had: the token permission table, the GitHub configuration, why a landed pull request shows as *Merged*, wiring real GitHub Actions, and the GitHub-specific failure modes. Its provider-neutral operational material — `land-list` / `land-watch`, `SQ_TOKEN`, service logs, `QUEUE_LOG_LEVEL` — moves with it, into the document people read first. No test target is mentioned in any of it.

**`make demo-requests`** (was `demo-pr`, in `service/submitqueue/demo/requests`) works without GitHub. `createOne` was the only function that touched the provider; it now describes a change and hands it to a `changeSource`. Three implement it: `fakeSource` (no I/O at all — mints a reproducible `git://` URI), `gitSource` (pushes real branches with the pinned `@git//:git`, serializing its commands because one working tree cannot take concurrent checkouts), and `githubSource` (the REST client, extracted unchanged). `GITHUB_TOKEN` is read only when it is needed.

**Folder overlap, made real.** A change writes all of its files into one folder under `demo/`, and states the paths it touched on its change URI (`sq-files=`, beside the existing `sq-fake=`) for the fake change provider to report back. All three profiles then use `pathoverlap` by directory, so a run shows both halves of the queue's behaviour: changes sharing a folder are batched in order and speculate on each other, changes in different folders go out beside each other.

How many folders there are is `FOLDERS`, defaulting to a number between five and ten picked per run and printed with the run's opening line. It is the dial on what a run demonstrates: `FOLDERS=1` serializes everything, a number well above `COUNT` keeps everything apart. The previous layout could do neither — it hashed each file's own name into two nested levels of 256, which put a twelve-file run's odds of any collision at roughly one in a thousand and scattered a single change across as many directories as it had files. What changed is the bucket count and what gets hashed, not the naming.

**Matching stack lifecycle targets.** `local-submitqueue-stop` did not exist — the only way to stop the stack was `local-stop`, which stops every domain's. It exists now, and `local-submitqueue-clean` removes the git sandbox and the overlay's volumes rather than only the base file's, which is what the documentation already claimed it did.

**`tool/gitsandbox`** provisions the bare repository `PROVIDER=git` merges into, idempotently, so a restart keeps whatever landed. **`platform/gitexec`** locates git and strips the ambient environment, so a developer's hooks or signing key cannot fail a demo.

Three fixes fell out of getting this to work at all:

- Runway's checkout directory moves to a named volume unless a path is given. It is where git clones, cherry-picks and commits, and on macOS a freshly written loose object read back over a bind mount can report `loose object … is corrupt` — which failed the first land against every new stack. The E2E still passes a path and still gets a bind mount.
- The Runway image now creates `/var/runway/checkouts`, so a named volume mounted there starts with a mode the service can write. Without it the service failed at boot with `mkdir: permission denied` whenever it ran as a non-root user.
- Change URIs are storage keys capped at 255 bytes, so the file marker names one path per directory rather than every file — for a directory-keyed analyzer that is the same set of keys — and drops paths that would not fit rather than truncating one into a different directory.

## Test Plan

Every command in the rewritten quickstart was run by hand against a live stack, and the output quoted in it is what it printed.

- ✅ `make local-submitqueue-start` (defaults to `fake`) then `make demo-requests COUNT=6` — all six landed
- ✅ inspected `batch_dependent` across three runs. `FOLDERS=1` produced a full chain — every batch depending on every later one. `FOLDERS=50` produced four batches with no dependents at all. The default picked eight folders, put two of six changes in the same one, and recorded a dependency between exactly those two and nothing else
- ✅ `make local-submitqueue-stop` leaves the sandbox and its data; `make local-submitqueue-clean` removes the sandbox, the volumes and the images, verified by looking for leftovers afterwards
- ✅ `PROVIDER=git make local-submitqueue-start` from a clean slate, then `PROVIDER=git make demo-requests` — real branches and multi-file commits, verified with `git -C /tmp/sq-sandbox/sandbox.git log --oneline main`
- ✅ `PROVIDER=git make demo-requests STACKED=true` — three changes, exactly one more entry in the target's reflog, so the stack landed atomically
- ✅ first land against a brand-new stack, repeatedly, which is the case the named volume fixes
- ✅ the 255-byte limit is a real one this hit first: an eight-file change was rejected with `change URI exceeds 255 bytes` before the marker was budgeted. Covered now by a unit test that builds a URI from `changeFilePath` output and asserts both the length and that it still parses
- ✅ `make test` (105 targets), `make lint`, `make check-gazelle`, `make check-tidy`
- ✅ `make e2e-test` and `make e2e-git-test`, after the rename and again after the compose change

`PROVIDER=github` is unchanged in behaviour but needs a token, so it has not been re-run; the GitHub source is the previous code path moved behind the interface.

## Note for reviewers

`make local-provider-start` is gone — use `make local-submitqueue-start`, which with no arguments now means `fake` where the old provider target meant GitHub. `make local-stop` still stops every domain's stack; `make local-submitqueue-stop` is the new one that stops only this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants