Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/adr/0002-decision-model-and-rl-environment.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,12 @@ The list leaves out models above 10 billion parameters and models built on a bas

A trained model replaces the base model only when a repository administrator approves it. The evaluation report gives them two numbers to compare against the base model on the test set: the mean reward, which should be higher, and the calibration error, which should not be worse. This version writes the report, but Hub does not record the decision or serve a model yet.

The customer-operated local reference runtime records that decision outside Hub. It accepts a taskset, tracks a job bounded by update and wall-time limits, writes the adapter and evaluation report as an immutable candidate, and serves only the candidate an operator explicitly promotes. A deterministic fixture backend runs this lifecycle without a GPU or external model download. Fixture candidates are for development and tests only and cannot be promoted in production. The runtime keeps a backend interface for a later real-model implementation without adding one here.

## Consequences

- Teams can train on their own review outcomes without sending them anywhere they did not choose.
- Teams can practice and test the complete lifecycle with a deterministic backend before spending GPU budget. Its report proves the orchestration and scoring path, not real-model quality.
- A taskset copies screenshots and learning text. It lives under `.replica/` in a checkout, or outside any checkout. When someone withdraws training consent, `replica-learning` lists the export the taskset came from, and the owner must delete the taskset too.
- The Builds file holds screenshots of every published Build of each reviewed prototype. The operator should delete it after writing the taskset.
- A "follows" label rests on the reviewer's check of the fix, which may have covered one width and theme. The other screenshots of that page count as fixed unless a later thread reports the same theme there.
Expand Down
2 changes: 2 additions & 0 deletions packages/learning/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,8 @@ A task shows every screenshot of its page, desktop before narrow and light befor

Hub names each screenshot by the address the build action was asked to capture. The build action follows redirects and Controls URL defaults without recording where the page ended up, so a comment made at the address a page redirects to finds no screenshot.

The [local decision-model runtime](environments/brand-check/README.md#run-the-local-training-and-inference-runtime) accepts that taskset, tracks a bounded deterministic fixture job, stores an immutable candidate and proper-scoring evaluation report, requires explicit promotion or rollback, and serves the promoted result through an authenticated scoring endpoint. The fixture proves the local lifecycle without a GPU or model download, but its candidates are development/test-only and cannot be promoted in production. The runtime keeps a small backend interface for later real-model work.

The output directory must be new and, inside a checkout, under `.replica/`. It holds:

| File | Contents |
Expand Down
100 changes: 100 additions & 0 deletions packages/learning/environments/brand-check/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,106 @@ The command prints a JSON report:

A missing or invalid answer counts as wrong for accuracy and as `invalid`, and it is left out of the other scores. A metric without data is `null`.

## Run the local training and inference runtime

The local runtime proves one complete customer-operated lifecycle. It keeps mutable job records under `jobs/`, writes each completed candidate once under `candidates/`, keeps immutable promotion generations under `promotions/`, and serves only the candidate an operator explicitly promotes or restores. Replica does not receive the taskset, adapter, reports, token, or requests.

The only backend in this first reference is deterministic and ignores image content. It uses the real taskset loader, proper-scoring evaluation, candidate store, promotion gate, rollback, and inference contract without a GPU or model download. Its candidates are for development and tests only; the runtime refuses to promote them when `--environment production`.

```sh
cd packages/learning/environments/brand-check
python3 examples/runtime_demo.py --out /tmp/replica-runtime-demo
```

The output shows two bounded jobs, immutable candidates, baseline and candidate test reports, two promotions, an explicit rollback, and one structured Build evaluation from the restored candidate. `candidate.evaluation.evidence.meanRewardDelta` states the fixture's measured gain over the 50% baseline, not real visual-model quality. The demo leaves its state in `/tmp/replica-runtime-demo/runtime` so an operator can inspect it.

For a taskset created by `replica-learning decision taskset`, make a runtime token, install the package, and start the service:

```sh
(umask 077 && python3 -c 'import secrets; print(secrets.token_urlsafe(32))' > ~/.replica-runtime-token)
export REPLICA_RUNTIME_TOKEN="$(cat ~/.replica-runtime-token)"
uv venv --python 3.12
uv pip install -e .
.venv/bin/replica-decision-runtime \
--environment development \
--taskset /path/to/taskset --state /path/to/runtime-state --port 8010
```

Submit and track a bounded fixture job, then promote its candidate:

```sh
curl -sS -H "Authorization: Bearer $REPLICA_RUNTIME_TOKEN" \
-H "Content-Type: application/json" \
-d '{"backend":"fixture","maxUpdates":12,"maxSeconds":30}' \
http://127.0.0.1:8010/v1/jobs

curl -sS -H "Authorization: Bearer $REPLICA_RUNTIME_TOKEN" \
http://127.0.0.1:8010/v1/jobs/<job id>

curl -sS -H "Authorization: Bearer $REPLICA_RUNTIME_TOKEN" \
-H "Content-Type: application/json" \
-d '{"candidateId":"<candidate id>","promotedBy":"<operator>"}' \
http://127.0.0.1:8010/v1/promotions

curl -sS -H "Authorization: Bearer $REPLICA_RUNTIME_TOKEN" \
-H "Content-Type: application/json" \
-d '{"rolledBackBy":"<operator>"}' \
http://127.0.0.1:8010/v1/rollbacks
```

`POST /score` and `POST /v1/evaluations` accept a task-shaped request with `state`, image data URIs, and `questions`. Send the runtime token as a bearer token. The fixture ignores the screenshots, so do not use its answers to judge a Build or guide authoring. The response includes probability `answers`, the promoted candidate ID, promotion generation, and one `followsProbability` and `breaksProbability` pair per learning. Inference returns `409 candidate_not_promoted` until an operator promotes a candidate.

```sh
curl -sS -H "Authorization: Bearer $REPLICA_RUNTIME_TOKEN" \
-H "Content-Type: application/json" \
--data-binary @request.json \
http://127.0.0.1:8010/score
```

The HTTP contract stays independent of the local store and trainer implementation:

| Route | Result |
| --- | --- |
| `GET /health` | Process liveness; no token required |
| `GET /ready` | Returns 200 only when an intact promoted candidate has a configured backend; no token required |
| `POST /v1/jobs` | Enqueues one job bounded by `maxUpdates` and `maxSeconds` |
| `GET /v1/jobs/<id>` | Returns `queued`, `running`, `succeeded`, or `failed` |
| `GET /v1/candidates/<id>` | Returns the immutable manifest and evaluation report |
| `POST /v1/promotions` | Moves the explicit promotion pointer to one candidate |
| `POST /v1/rollbacks` | Restores the immediately previous candidate as a new promotion generation |
| `GET /v1/promotion` | Returns the promoted candidate and generation |
| `POST /score`, `POST /v1/evaluations` | Evaluates one page with the promoted candidate |

### Run the fixture runtime in a container

Build the runtime image, mount the taskset read-only, and mount a separate durable directory read-write:

```sh
docker build -f server/runtime.Dockerfile -t replica-decision-runtime:dev .
mkdir -p /path/to/runtime-state
docker run --rm \
-e REPLICA_RUNTIME_TOKEN \
-v /path/to/taskset:/data/taskset:ro \
-v /path/to/runtime-state:/data/runtime \
-p 127.0.0.1:8010:8010 \
replica-decision-runtime:dev
```

The image contains no taskset, candidate, model, or credential. It runs as user 10001, which needs read access to the taskset and write access to the runtime-state directory. Add `--user "$(id -u):$(id -g)"` when the mounted directories allow only your account; rootless Podman also needs `--userns=keep-id`.

### Add another backend later

`TrainingBackend` is the extension seam. A backend can run in a separate terminable process, declares its name and whether its candidates may be promoted in production, implements one bounded `train` call that returns an adapter plus baseline and candidate reports, and implements `predict` for that adapter. Installed runtimes use a spawned worker process, so a backend and its configuration must be serializable. The runtime owns job state, candidate integrity, promotion, rollback, and HTTP authentication. This first reference does not connect the existing d1 trainer to that interface.

### Trust, data, and credential boundaries

- The operator chooses the host. The local runtime never calls Replica or a cloud service. A later cloud implementation can keep the same job, candidate, promotion, and inference contracts.
- The taskset contains private screenshots, learning text, and labels. Mount it read-only. A training-consent withdrawal still requires the operator to delete the taskset and every candidate derived from that export.
- The runtime state contains adapters, evaluation reports, job errors, and the promotion pointer. Back it up or delete it under the customer's retention rules. Candidate files are created read-only and never updated in place.
- The runtime token authorizes training, promotion, candidate reads, and inference. Keep it in a protected file or secret store and pass it through the environment, not the command line or image.
- The service uses plain HTTP. Bind it to localhost as shown, or put customer-operated TLS and authentication in front of it before another host can connect.
- A backend is trusted code. The local runtime checks the taskset first and bounds time and updates, but a later model backend can read the screenshots and return candidate artifacts.

## Train d1-omni-600M

[examples/train_d1_omni.py](examples/train_d1_omni.py) trains Liquid AI's d1-omni-600M against this environment with reinforcement learning. It keeps the released model frozen and, by default, trains a small readout of 1,025 numbers on top of it. The readout reads the model's internal state at each yes-or-no question's answer options and corrects the released model's answer. `--part head` trains the whole decision head instead, about 26 million of the model's 587 million parameters. On a synthetic taskset of 518 practice pages, the head's dev score rose from 0.685 to at best 0.753, barely above the 0.750 that answering 50% to every question earns, while its practice score rose much more. The readout raised the dev score to 0.807. Either way, the trainer learns from the rewards alone. Every 10 updates it scores the dev split and keeps the weights with the best dev reward, because a small taskset is easy to memorize.
Expand Down
13 changes: 12 additions & 1 deletion packages/learning/environments/brand-check/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,25 @@

from .taskset import question_options

__all__ = ["BrandCheckAction", "BrandCheckEnv", "BrandCheckObservation", "question_options"]
__all__ = [
"BrandCheckAction",
"BrandCheckEnv",
"BrandCheckObservation",
"FixtureBackend",
"LocalRuntime",
"RuntimeFailure",
"question_options",
]

# The scoring modules use only the standard library. Importing the OpenEnv classes on first use
# lets them, and their tests, run where openenv-core is not installed.
_OPENENV_EXPORTS = {
"BrandCheckAction": ".models",
"BrandCheckObservation": ".models",
"BrandCheckEnv": ".client",
"FixtureBackend": ".runtime",
"LocalRuntime": ".runtime",
"RuntimeFailure": ".runtime",
}


Expand Down
163 changes: 163 additions & 0 deletions packages/learning/environments/brand-check/examples/runtime_demo.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
"""Run the complete local training, promotion, and inference lifecycle without a model download."""

from __future__ import annotations

import argparse
import base64
import hashlib
import importlib.util
import json
import shutil
import sys
import time
from pathlib import Path

try:
from replica_brand_check.runtime import LocalRuntime
except ModuleNotFoundError:
root = Path(__file__).resolve().parent.parent
spec = importlib.util.spec_from_file_location(
"replica_brand_check", root / "__init__.py", submodule_search_locations=[str(root)]
)
assert spec is not None and spec.loader is not None
package = importlib.util.module_from_spec(spec)
sys.modules["replica_brand_check"] = package
spec.loader.exec_module(package)
from replica_brand_check.runtime import LocalRuntime

PNG = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNkYAAAAAYAAjCB0C8AAAAASUVORK5CYII="
)
LABELS = {
"mem_111111111111111111111111": (
"Put a visible label above every form field.",
True,
),
"mem_222222222222222222222222": (
"Keep destructive actions away from the primary action.",
False,
),
}


def task(number: int, split: str) -> dict:
questions = {
key: {
"type": "noul",
"instructions": "Does the candidate follow this team learning?",
"criteria": {"true": "It follows the learning", "false": "It breaks the learning"},
}
for key in LABELS
}
return {
"schemaVersion": "replica.decision-task/v1",
"id": f"task_{number:024x}",
"split": split,
"state": {
"rules": [{"id": key, "text": text} for key, (text, _) in LABELS.items()],
"imagesNote": "Image 1 is the candidate: route / at desktop width in light theme.",
},
"images": [
{
"role": "candidate",
"path": "images/screen.png",
"mediaType": "image/png",
"digest": "sha256:" + hashlib.sha256(PNG).hexdigest(),
"size": "desktop",
"theme": "light",
}
],
"questions": questions,
"labels": {key: {"answer": answer} for key, (_, answer) in LABELS.items()},
"provenance": {
"exportId": "export_fixture",
"prototypeId": number,
"buildNumber": 1,
"route": "/",
},
}


def write_taskset(directory: Path) -> None:
directory.mkdir(parents=True)
(directory / "images").mkdir()
(directory / "images" / "screen.png").write_bytes(PNG)
tasks = [
task(1, "train"),
task(2, "train"),
task(3, "train"),
task(4, "dev"),
task(5, "test"),
task(6, "test"),
]
manifest = {
"schemaVersion": "replica.decision-taskset/v1",
"exportId": "export_fixture",
"createdAt": "2026-11-10T00:00:00Z",
"counts": {"tasks": len(tasks)},
}
(directory / "taskset.json").write_text(json.dumps(manifest), encoding="utf-8")
(directory / "tasks.jsonl").write_text(
"".join(json.dumps(item) + "\n" for item in tasks), encoding="utf-8"
)


def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--out", required=True, type=Path, help="new directory for demo data")
args = parser.parse_args()
if args.out.exists():
print(f"error: {args.out} already exists; choose a new demo directory.", file=sys.stderr)
return 1
taskset = args.out / "taskset"
state = args.out / "runtime"
write_taskset(taskset)
runtime = LocalRuntime(state, taskset)
try:
jobs = []
for updates in (4, 8):
submitted = runtime.submit(
backend="fixture", max_updates=updates, max_seconds=30
)
while True:
job = runtime.job(submitted["id"])
if job["state"] in ("succeeded", "failed"):
break
time.sleep(0.01)
if job["state"] != "succeeded":
raise RuntimeError(job["error"]["message"])
jobs.append(job)
first = runtime.promote(jobs[0]["candidateId"], promoted_by="demo-operator")
second = runtime.promote(jobs[1]["candidateId"], promoted_by="demo-operator")
rollback = runtime.rollback(rolled_back_by="demo-operator")
candidate = runtime.candidate(rollback["candidateId"])
encoded = base64.b64encode(PNG).decode()
response = runtime.evaluate(
{
"state": task(7, "test")["state"],
"images": [f"data:image/png;base64,{encoded}"],
"questions": task(7, "test")["questions"],
}
)
print(
json.dumps(
{
"jobs": jobs,
"candidate": candidate,
"promotions": [first, second, rollback],
"evaluation": response,
"stateDirectory": str(state.resolve()),
},
indent=2,
)
)
except Exception:
shutil.rmtree(args.out, ignore_errors=True)
raise
finally:
runtime.close()
return 0


if __name__ == "__main__":
sys.exit(main())
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ dependencies = ["openenv-core==0.3.0"]

[project.scripts]
server = "replica_brand_check.server.app:main"
replica-decision-runtime = "replica_brand_check.runtime_server:main"

[tool.setuptools]
packages = ["replica_brand_check", "replica_brand_check.server"]
Expand Down
Loading
Loading