feat: add Vercel Sandbox compute backend by DarlingtonDeveloper · Pull Request #832 · alphaXiv/OpenResearch

DarlingtonDeveloper · 2026-10-30T15:25:57Z

Closes #831

Summary

Adds --backend vercel: runs an experiment in a Vercel Sandbox microVM on the user's own Vercel project. A bundled launcher drives the official vercel Python SDK from an orx-managed venv (src/jobs/vercel.rs, src/local/vercel.rs).
Sandboxes are addressed by name (orx-<run id>), so a restarted orx supervise reattaches. If the sandbox is already gone (HTTP 404), status reports DELETED and the run is finalised instead of staying running.
SDK process output streams are single-use, so the job writes to a log file in the sandbox and logs replays it from line 1. GNU tail --pid is used when the image has it. A custom image without it (busybox/alpine) falls back to a polling loop, and an image that cannot read the log at all surfaces an error instead of an empty log.
The Sandbox API records a process's exit only once a client waits on it. Status checks the job's pid inside the sandbox first and waits only after it has exited.
A sandbox outlives its command until the session limit, so once a run is terminal and its log is drained, the supervisor stops and destroys the sandbox. Cancel stops it and the run is reported as cancelled.
Credentials: VERCEL_OIDC_TOKEN, or VERCEL_TOKEN + VERCEL_TEAM_ID + VERCEL_PROJECT_ID, from the process env or Settings → Environment. They are stripped from the job's env. Other synced secrets are passed in a 0600 file the job sources, not in sandbox or process config.
Settings → Compute → Vercel Sandbox is a detail page (not a quick-setup dialog, because there is nothing to save there). It shows each variable's source (process environment, or a link to the Environment tab), the region (ORX_VERCEL_REGION or the project default, iad1 when the project has none), and whether the managed SDK venv is installed.
Vercel is the newest backend, so it is last in the target lists, just before openresearch.
--flavor is an optional vCPU count (cpu = 2, cpu-<n>, <n>; 1 or an even number up to 32). It is CPU only. When VERCEL_TOKEN and VERCEL_TEAM_ID are set, one GET /v2/teams/{teamId} at launch reads billing.plan. Hobby caps vCPUs at 4 and Pro at 8; Enterprise keeps 32. Current Vercel docs match those numbers (https://vercel.com/docs/sandbox/pricing and the Sandbox FAQ). OIDC-only auth, or a failed lookup, does not block the launch; a plan limit then shows up on sandbox create.
Timeout defaults to 4 hours, or 40 minutes on Hobby. A session is at most 45 minutes on Hobby and 24 hours on Pro and Enterprise. orx keeps 5 minutes after the job for the exit status and the last log lines, so --timeout can be at most 40m on Hobby and 23h55m otherwise. A longer value is rejected before submit when the plan is known. The default image is vercel/sandbox/python:3.14. ORX_VERCEL_REGION optionally picks a region.
Registered in compute status, show and test, the exp run help text, the dashboard target lists (all locales), and the orx-compute agent skill (references/vercel.md).

Test plan

cargo fmt --all --check
cargo clippy --all-targets --locked -- -D warnings
cargo test --locked: 1322 unit, 1 build identity, 19 CLI. New tests cover Hobby/Pro/Enterprise timeout and vCPU caps (including the 5-minute grace), team billing.plan parsing and a local GET /v2/teams/{id} lookup, a 404 status mapping to DELETED, the portable log follow (GNU tail and a stub that rejects --pid), and a clear error when tail cannot read the log.
ui: check-i18n, check-styles, vite build, pnpm typecheck, node --test ui/tests (545 pass, including the Vercel settings page test)
Manual, against a real Vercel Pro project (previous commit): a run completed with streamed logs and no Vercel credentials in the job env; --flavor 4 gave 4 CPUs; the run was marked done about 6s after the job finished and the sandbox was destroyed; supervisor reattach completed a run already in flight; orx exp cancel stopped the sandbox and recorded the run as cancelled. All test sandboxes were confirmed deleted (404).
Not verified against a real Hobby team. The plan lookup, 40-minute default, and pre-submit rejection are covered with a fake HTTP response and unit tests. A Hobby account would still be the check that billing.plan is hobby and that Vercel rejects a session over 45 minutes when the plan cannot be read.

Open question: does the API's analytics ingest accept "vercel" as a computeTarget value?

Adds `--backend vercel`, running experiments in a Vercel Sandbox microVM on the user's own Vercel project, driven through the official `vercel` Python SDK in an orx-managed venv (same shape as the Modal backend). - Submit uploads the source snapshot and starts the job; sandboxes are addressed by name so a restarted `orx supervise` reattaches. - The job logs to a file in the sandbox that `logs` replays from line 1, since SDK process streams are single-use. - The Sandbox API records a process exit only once a client waits on it, so status probes the job's pid first and waits only after it has exited. - Sandboxes outlive their command until the session limit, so the supervisor stops and destroys them once the run is terminal. - Credentials: VERCEL_OIDC_TOKEN, or VERCEL_TOKEN + VERCEL_TEAM_ID + VERCEL_PROJECT_ID, from the env or Settings > Environment. They are stripped from the job env; other secrets ride a 0600 file, not sandbox config. - `--flavor` is an optional vCPU count (cpu, cpu-<n>, <n>); CPU only. Default image vercel/sandbox/python:3.14, 4h default timeout, 24h cap. - Registers the target in compute status/show/test, the dashboard target lists (label, description, logo, flavor hints, all locales), and the orx-compute agent skill. Verified end to end against a real Vercel project: completion, log streaming, 4-vCPU flavor, supervisor reattach, cancel, and sandbox cleanup.

greptile-apps · 2026-10-30T15:30:05Z

Confidence Score: 4/5

[High impact] The change adds a user-selected route for experiment code and job variables to run on a Vercel project.

Fix final log collection before merging so sandbox cleanup cannot delete unread experiment output.

Findings

Final logs can disappear ▶
Saved credentials are ignored ▶
Failed cleanup never retries ▶

Fix with agent prompt

### Issue 1
src/commands/supervise.rs:698-702
After a supervisor restart or connection outage, replaying a large log can take more than 20 seconds. This path aborts the reader at that point, then calls `finish_job()` to destroy the sandbox. Any unread output is permanently lost. Keep the sandbox until the log has been collected, or save the unfinished work for a later retry.

### Issue 2
src/jobs/vercel.rs:300-301
`credential()` falls back to Settings > Environment when a process variable is empty, but `auth_env()` only fills variables that are absent. With `VERCEL_TOKEN=""` and a valid saved token, readiness passes but the launcher inherits the empty token and cannot launch the run. Use the same credential resolver for readiness and the launcher.

### Issue 3
src/commands/supervise.rs:714-721
The run is already terminal when `finish_job()` runs. If stopping the sandbox fails temporarily, this branch only prints the error and exits. Automatic recovery and manually restarted supervisors both skip terminal runs, so the sandbox can keep billing until its session limit. Save cleanup as pending work, or retry it before exiting.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds Vercel Sandbox as a CPU-only experiment backend.

Local experiments can run in a Vercel Sandbox on the user's project.
The supervisor follows Vercel jobs through logs, cancellation, and cleanup.
Vercel appears in compute choices and backend guidance.

Acknowledged scope: DarlingtonDeveloper explicitly deferred a dedicated settings card and raised analytics acceptance of vercel as an open question.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[CLI or dashboard] --> B[Save source snapshot and reserve run]
  B --> C[Python launcher creates Vercel sandbox]
  C --> D[Upload source and private environment file]
  D --> E[Start experiment and write remote log]
  E --> F[Detached supervisor polls status and replays logs]
  F --> G[Save final result locally]
  G --> H[Stop and destroy sandbox]

_{Reviews (1) · Last reviewed commit: "feat: add Vercel Sandbox compute backend" · Reviewed by Greptile}

greptile-apps · 2026-10-30T15:30:09Z

+            if tokio::time::timeout(Duration::from_secs(20), &mut log_task)
+                .await
+                .is_err()
+            {
+                log_task.abort();


Final logs can disappear

After a supervisor restart or connection outage, replaying a large log can take more than 20 seconds. This path aborts the reader at that point, then calls finish_job() to destroy the sandbox. Any unread output is permanently lost. Keep the sandbox until the log has been collected, or save the unfinished work for a later retry.

Knowledge Base Used: Remote jobs, hosts, and supervision

Prompt To Fix With AI

This is a comment left during a code review. Path: src/commands/supervise.rs Line: 698-702 Comment: **Final logs can disappear** After a supervisor restart or connection outage, replaying a large log can take more than 20 seconds. This path aborts the reader at that point, then calls `finish_job()` to destroy the sandbox. Any unread output is permanently lost. Keep the sandbox until the log has been collected, or save the unfinished work for a later retry. **Knowledge Base Used:** [Remote jobs, hosts, and supervision](https://app.greptile.com/alphaxiv/-/custom-context/knowledge-base/alphaxiv/openresearch/-/docs/remote-job-supervision.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Valid, fixed in the latest commit. The fixed 20s drain is replaced by drain_vercel_logs: it keeps waiting while the run log is still being written (a fresh replay truncates it first, so any size change counts as progress) and abandons the replay only after 45s with no writes, which is longer than one stream pass's 30s idle window. The sandbox is destroyed only after that.

greptile-apps · 2026-10-30T15:30:09Z

+        .filter(|key| std::env::var_os(key).is_none())
+        .filter_map(|key| crate::config::synced_env_var(key).map(|v| (key.to_string(), v)))


Saved credentials are ignored

credential() falls back to Settings > Environment when a process variable is empty, but auth_env() only fills variables that are absent. With VERCEL_TOKEN="" and a valid saved token, readiness passes but the launcher inherits the empty token and cannot launch the run. Use the same credential resolver for readiness and the launcher.

Knowledge Base Used: Compute backend adapters

Prompt To Fix With AI

This is a comment left during a code review. Path: src/jobs/vercel.rs Line: 300-301 Comment: **Saved credentials are ignored** `credential()` falls back to Settings > Environment when a process variable is empty, but `auth_env()` only fills variables that are absent. With `VERCEL_TOKEN=""` and a valid saved token, readiness passes but the launcher inherits the empty token and cannot launch the run. Use the same credential resolver for readiness and the launcher. **Knowledge Base Used:** [Compute backend adapters](https://app.greptile.com/alphaxiv/-/custom-context/knowledge-base/alphaxiv/openresearch/-/docs/compute-backends.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Valid, fixed. Readiness and the launcher env now share one resolver (resolve_credential): a process value counts only when non-empty, otherwise the saved Settings > Environment value is used, and the launcher explicitly sets any key resolved from the saved env. A unit test covers an empty VERCEL_TOKEN in the process env with a saved token.

greptile-apps · 2026-10-30T15:30:09Z

+            if let Err(err) = vercel::finish_job(&name).await {
+                eprintln!(
+                    "supervise {run_id}: could not stop vercel sandbox {name} \
+                     (it stops at its session limit): {err}"
+                );
+            }
+            eprintln!("supervise {run_id}: finished ({status})");
+            return Ok(());


Failed cleanup never retries

The run is already terminal when finish_job() runs. If stopping the sandbox fails temporarily, this branch only prints the error and exits. Automatic recovery and manually restarted supervisors both skip terminal runs, so the sandbox can keep billing until its session limit. Save cleanup as pending work, or retry it before exiting.

Knowledge Base Used: Compute backend adapters

Prompt To Fix With AI

This is a comment left during a code review. Path: src/commands/supervise.rs Line: 714-721 Comment: **Failed cleanup never retries** The run is already terminal when `finish_job()` runs. If stopping the sandbox fails temporarily, this branch only prints the error and exits. Automatic recovery and manually restarted supervisors both skip terminal runs, so the sandbox can keep billing until its session limit. Save cleanup as pending work, or retry it before exiting. **Knowledge Base Used:** [Compute backend adapters](https://app.greptile.com/alphaxiv/-/custom-context/knowledge-base/alphaxiv/openresearch/-/docs/compute-backends.md) --- For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Valid, fixed. finish_vercel retries stop and destroy 5 times with backoff (2, 4, 8 and 16s) before giving up, and the launcher's finish now treats an already-deleted sandbox (404) as success, so a retry after a partial success doesn't fail. Retrying across supervisor restarts would need a pending-cleanup record on terminal runs, which no backend has today, so I left that out of scope. If it's still missed, the sandbox stops at its session limit (the job deadline plus 5 minutes).

- Drain the final log replay while the run log keeps changing instead of a fixed 20s, abandoning it only after 45s without writes, so destroying the sandbox cannot drop unread output. - Resolve credentials once for readiness and the launcher env: an empty process value falls back to the saved Settings > Environment value. - Retry stopping and destroying a finished run's sandbox with backoff, and treat an already-deleted sandbox as success.

paperclover

my main issue is there is a page that you can reach that's just blank. regardless if this is an issue with existing compute providers, it should not be made worse. please complete the pr.

i also intend to change this compute menu a bit later, but in the meantime, new providers should go at the end.

ai powered code notes:

Settings page. Clicking the Vercel Sandbox tile opens an empty page with just the logo and title, because vercel isn't in QUICK_SETUP_TARGETS and BackendDetailPage has no section for it. Could you build the page out? Like Modal or HF, it should show where the credentials come from (process env vs. the Environment tab, with a link), the region, and whether the managed SDK env is ready.

Order. Vercel sits in the middle of the backend list (between Modal and Kubernetes), both in the UI and in the code. Since it's the newest backend, please move it to the end, just before openresearch. That means the targets in compute_settings_json (which sets the tile order), plus BACKENDS, FLAVORED_BACKENDS, the Backend enum, the backend() match, ComputeTargetId, the target maps in SettingsPage.tsx / computeTargets.ts, the message files, and the skill/help text lists.

Hobby plan. The default 4h timeout becomes a 4h05m session, but Hobby caps sessions at 45 minutes, so I'd expect a Hobby launch without --timeout to fail when the sandbox is created. With VERCEL_TOKEN + VERCEL_TEAM_ID, GET /v2/teams/{teamId} returns billing.plan (hobby | pro | enterprise). One call at launch could set the default timeout to 45m on Hobby, reject a longer --timeout before submitting, and cap vCPUs at 4 on Hobby and 8 on Pro. The OIDC-only path probably can't read the team, so it can fall back to a clear error.

A few smaller things from reading the code:

If the sandbox is gone (404) when a supervisor reattaches, the launcher's status throws and the supervisor retries forever, so the run stays running. finish already treats 404 as done; status should report a terminal stage too, the way Modal maps NotFound to DELETED.

With --timeout 24h, the session limit equals the job's kill deadline, so the 5-minute grace disappears. The run then ends as "sandbox stopped before the job exited", with no exit code and no final log lines. Capping --timeout at 24h minus the grace (or documenting it) would avoid that.

Logs depend on GNU tail --pid. A custom --image without it (busybox/alpine) returns no logs and shows no error.

The Settings compute tile opened a blank page, and a Hobby launch used a 4 hour timeout the plan cannot run. Show where each credential comes from, the region, and whether the managed SDK environment is installed. List Vercel last, just before OpenResearch. Read the team plan once at launch when a token and team id are available, keep 5 minutes of the session for logs and the exit status, and treat a missing sandbox as deleted. Follow logs with a portable poll when the image has no GNU tail --pid. Co-authored-by: Mike <DarlingtonDeveloper@users.noreply.github.com>

DarlingtonDeveloper · 2026-10-31T02:40:43Z

@paperclover thanks for the review. All addressed in 5d6b872:

Settings page: The Vercel tile now opens a settings page. Each credential shows whether it comes from the process environment or the Environment tab (that one is a link), plus the region and whether the managed SDK environment is installed. I left it as a page rather than a quick-setup dialog because there is nothing to save on that screen.
Order: Vercel is now last in the backend lists, just before openresearch, in the UI, the Rust lists, the help text, and the skill.
Hobby: With VERCEL_TOKEN and VERCEL_TEAM_ID, launch does one GET /v2/teams/{teamId} and reads billing.plan. Current docs still say Hobby 45 minutes / 4 vCPUs, Pro 24 hours / 8, Enterprise 24 hours / 32, so those are the caps. Hobby's default job timeout is 40 minutes so the session, including the 5-minute grace, stays inside 45 minutes. A longer --timeout or too many vCPUs is rejected before submit. OIDC-only and a failed lookup do not block; the create error names the cap.
404: status now reports DELETED on a 404, so supervise finalises the run instead of retrying.
Grace: --timeout is capped at the session limit minus 5 minutes (23h55m, or 40m on Hobby), and the error says why.
Logs: The sandbox uses GNU tail --pid when the image has it, and otherwise polls the log until the pid exits. If tail cannot read the file, supervise gets an error instead of an empty log. Both are documented in references/vercel.md.

greptile-apps Bot reviewed Oct 30, 2026

View reviewed changes

paperclover requested changes Oct 30, 2026

View reviewed changes

Merge branch 'main' into feat/vercel-sandbox-backend

7cd00e4

Navigation Menu

Repository navigation

feat: add Vercel Sandbox compute backend - #832

feat: add Vercel Sandbox compute backend#832
DarlingtonDeveloper wants to merge 4 commits into
alphaXiv:mainfrom
DarlingtonDeveloper:feat/vercel-sandbox-backend

DarlingtonDeveloper commented Oct 30, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 30, 2026 •

edited

Loading

Diagram

Uh oh!

greptile-apps Bot Oct 30, 2026

Uh oh!

DarlingtonDeveloper Oct 30, 2026

Uh oh!

greptile-apps Bot Oct 30, 2026

Uh oh!

DarlingtonDeveloper Oct 30, 2026

Uh oh!

greptile-apps Bot Oct 30, 2026

Uh oh!

DarlingtonDeveloper Oct 30, 2026

Uh oh!

paperclover left a comment

Uh oh!

DarlingtonDeveloper commented Oct 31, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		.filter(\|key\| std::env::var_os(key).is_none())
		.filter_map(\|key\| crate::config::synced_env_var(key).map(\|v\| (key.to_string(), v)))

Conversation

DarlingtonDeveloper commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

Summary

Diagram

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

DarlingtonDeveloper Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

DarlingtonDeveloper Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

DarlingtonDeveloper Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

paperclover left a comment

Choose a reason for hiding this comment

Uh oh!

DarlingtonDeveloper commented Oct 31, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

DarlingtonDeveloper commented Oct 30, 2026 •

edited

Loading

greptile-apps Bot commented Oct 30, 2026 •

edited

Loading