Cadence scheduler with ingestion budgets and overflow folding

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +1,5 @@
 blank_issues_enabled: true
 contact_links:
   - name: Security reports
-    url: https://github.com/tinyhumansai/rust-template/security/policy
+    url: https://github.com/tinyhumansai/tinyeventqueues/security/policy
     about: Please do not report vulnerabilities through public issues.

.github/workflows/ci.yml

@@ -54,27 +54,27 @@ jobs:
         run: cargo test
 
       # `cargo build --all-targets` only *compiles* an example. `AGENTS.md`
-      # promises `cargo run -p template --example basic` works, and a compiled
+      # promises `cargo run -p tinyeventqueues --example basic` works, and a compiled
       # example can still fail on its first line.
       - name: Run the bundled example
-        run: cargo run -p template --example basic
+        run: cargo run -p tinyeventqueues --example basic
 
-      # `crates/template-bus` exists so a host can name the payload types
+      # `crates/tinyeventqueues-bus` exists so a host can name the payload types
       # without compiling the module. That promise is invisible in a diff,
       # because a forbidden dependency arrives transitively through a feature
       # someone enabled one crate away — so it is asserted rather than
       # documented.
       #
-      # The FORWARD form is required. `cargo tree -i <crate> -p template-bus`
+      # The FORWARD form is required. `cargo tree -i <crate> -p tinyeventqueues-bus`
       # discards the `-p` scope, prints the whole-workspace inverse tree, and
       # exits 0 looking clean even when this crate is the one at fault.
       - name: Assert the contract crate stays transport-free
         run: |
           set -euo pipefail
-          forbidden="$(cargo tree -p template-bus -e normal,build --prefix none \
+          forbidden="$(cargo tree -p tinyeventqueues-bus -e normal,build --prefix none \
             | grep -Ei 'tinybus|tokio|reqwest|ureq|hyper|rusqlite|git2' || true)"
           if [ -n "$forbidden" ]; then
-            echo "template-bus pulled in a dependency its manifest forbids:" >&2
+            echo "tinyeventqueues-bus pulled in a dependency its manifest forbids:" >&2
             echo "$forbidden" >&2
             echo >&2
             echo "The contract is what a host compiles against. It must stay free" >&2
@@ -128,7 +128,7 @@ jobs:
         run: |
           set -euo pipefail
           msrv="$(cargo metadata --format-version 1 --no-deps \
-            | jq -r '.packages[] | select(.name == "template") | .rust_version')"
+            | jq -r '.packages[] | select(.name == "tinyeventqueues") | .rust_version')"
           if [[ -z "$msrv" || "$msrv" == "null" ]]; then
             echo "workspace.package.rust-version is not set in Cargo.toml" >&2
             exit 1

.github/workflows/release.yml

@@ -22,9 +22,9 @@ permissions:
 
 env:
   # The workspace member that ships as the loadable module. Its package name is
-  # the artifact name and the library name; `crates/template-bus` rides along on
+  # the artifact name and the library name; `crates/tinyeventqueues-bus` rides along on
   # the same inherited version and is not packaged separately.
-  RELEASE_PACKAGE: template
+  RELEASE_PACKAGE: tinyeventqueues
 
 jobs:
   prepare:
@@ -259,7 +259,7 @@ jobs:
             macOS) module="target/release/lib${library_name}.dylib" ;;
             *) echo "unsupported Unix runner: ${RUNNER_OS}" >&2; exit 1 ;;
           esac
-          cargo run --locked --package template --example verify_module -- "$module"
+          cargo run --locked --package tinyeventqueues --example verify_module -- "$module"
 
       - name: Verify Windows module through TinyBus loader
         if: ${{ runner.os == 'Windows' }}
@@ -270,7 +270,7 @@ jobs:
           $ErrorActionPreference = 'Stop'
           $libraryName = $env:CRATE_NAME.Replace('-', '_')
           $module = "target/release/$libraryName.dll"
-          $verifyRoot = Join-Path $env:RUNNER_TEMP 'template-module-verify'
+          $verifyRoot = Join-Path $env:RUNNER_TEMP 'tinyeventqueues-module-verify'
           New-Item -ItemType Directory -Force $verifyRoot | Out-Null
 
           $identity = [System.Security.Principal.WindowsIdentity]::GetCurrent()
@@ -300,7 +300,7 @@ jobs:
 
           $verifiedModule = Join-Path $verifyRoot "$libraryName.dll"
           Copy-Item -LiteralPath $module -Destination $verifiedModule
-          cargo run --locked --package template --example verify_module -- $verifiedModule
+          cargo run --locked --package tinyeventqueues --example verify_module -- $verifiedModule
 
       - name: Assemble Unix module package
         if: ${{ runner.os != 'Windows' }}
@@ -463,7 +463,7 @@ jobs:
           verify_root="/opt/${CRATE_NAME}-module-verify"
           install -d -m 700 "$verify_root"
           install -m 755 "target/release/lib${library_name}.so" "$verify_root/"
-          cargo run --locked --package template --example verify_module -- \
+          cargo run --locked --package tinyeventqueues --example verify_module -- \
             "$verify_root/lib${library_name}.so"
 
       - name: Assemble distribution module package
@@ -587,5 +587,5 @@ jobs:
           cargo run --manifest-path vendor/tinybus/Cargo.toml --locked \
             --package tinybus --all-features --example github_module_host -- \
             "$release_url" "$archive" "$sha256"
-          cargo run --locked --package template --example verify_github_release -- \
+          cargo run --locked --package tinyeventqueues --example verify_github_release -- \
             "$release_url" "$archive" "$sha256"

AGENTS.md

@@ -4,63 +4,42 @@ This file is the single source of truth for how humans and coding agents work
 in this repository. `CLAUDE.md` is a symlink to this file, so every agent reads
 the same instructions.
 
-When you generate a new project from this template, keep this file and adapt
-the project-specific parts (crate name, module map, feature flags, commands).
-Delete guidance that no longer applies rather than leaving it to rot.
-
-## Template Checklist
-
-Do this once, in a single commit, before writing feature code:
-
-- [ ] Rename `crates/template` and `crates/template-bus` to the project's crate
-      names, and update `name` in each manifest plus the `template-bus` entry in
-      the root `[workspace.dependencies]`.
-- [ ] Set `description`, `keywords`, and `categories` in each manifest, and
-      `repository` in the root `[workspace.package]`.
-- [ ] Rename the crate references in `README.md`, both `src/lib.rs` files,
-      `crates/template/examples/`, and `crates/template/tests/` (search for
-      `template` and `template_bus`).
-- [ ] Replace the placeholder `greeting` module in both crates with the first
-      real feature area — payload types in the contract crate, behavior in the
-      module crate — keeping the `mod.rs` / `types.rs` / `test.rs` layout.
-- [ ] Confirm `license` and `LICENSE` match the project's intended license.
-- [ ] Update the security contact in `SECURITY.md`.
-- [ ] Rename the TinyBus interface, object path, and member constants in
-      `crates/template-bus/src/names/`, and the matching `provides` / `methods`
-      declarations in `crates/template/src/tinybus_module/`, while keeping
-      `vendor/tinybus` pinned.
-- [ ] Reset `CONTRACT_VERSION` in `crates/template-bus/src/version/` for the new
-      contract.
-- [ ] Replace `ROADMAP.md` with the real plan, or delete it.
-- [ ] Rewrite the "Project Structure" section below to describe this workspace.
+This repository is **tinyeventqueues**: external event management and
+queuing. It durably queues noisy inbound events and filters them before they
+reach an agent. The contract is `docs/specs/event-pipeline.md`; the delivery
+order is `docs/plans/event-pipeline.md`.
 
 ## Project Structure
 
 This is a Rust 2024 cargo workspace rooted at a virtual `Cargo.toml`. Every
-crate lives under `crates/`, one directory per package, each directory named for
-the package it holds. There is no root package: the crate that ships as the
-loadable module is `crates/template`, the same as any other member.
+crate lives under `crates/`, one directory per package.
 
 ```text
 Cargo.toml              # virtual workspace: members, [workspace.package],
                         # [workspace.dependencies], [workspace.lints]
+queues.json             # queue manifest: every queue and its spec (editable)
+pipeline.json           # pipeline tunables, queues by name (editable)
+schemas/                # JSON Schema for pipeline.json
 crates/
-├── template-bus/       # the wire contract: what crosses the bus, nothing else
-│   ├── README.md       # why the contract is its own crate
+├── tinyeventqueues-bus/   # the wire contract: what crosses the bus, nothing else
 │   └── src/
 │       ├── lib.rs      # crate docs + the entire public re-export surface
-│       ├── names/      # interface, object path, one constant per member
+│       ├── names/      # interface names, object paths, one constant per member
 │       ├── version/    # contract version and the host bind rule
-│       └── <family>/   # one directory per payload family
-└── template/           # the module: behavior, adapter, and the cdylib
+│       ├── event/      # the InboundEvent envelope
+│       ├── ops/        # member request/response payloads
+│       ├── host/       # host callback payloads (Turn, Deliver, ClassifySalience)
+│       ├── outcome/    # Disposition, Reason, Outcome
+│       └── config/     # PipelineConfig, the wire form of pipeline.json
+└── tinyeventqueues/       # the pipeline, the adapter, and the cdylib
     ├── src/
     │   ├── lib.rs      # crate docs + public surface, re-exporting the contract
-    │   ├── error/mod.rs      # crate-wide `Error` and `Result<T>`
-    │   ├── tinybus_module/   # TinyBus interface, ABI exports, integration tests
-    │   └── <feature>/        # one directory per feature area
-    │       ├── mod.rs        # module docs, wiring, smallest useful public API
-    │       ├── types.rs      # substantial type definitions
-    │       └── test.rs       # module-local unit tests
+    │   ├── error/      # crate-wide `Error` and `Result<T>`
+    │   ├── config/     # loading and validating queues.json / pipeline.json
+    │   ├── pipeline/   # `Pipeline` and `PipelineBuilder`
+    │   ├── <stage>/    # one directory per pipeline stage
+    │   ├── tinybus_module/   # TinyBus interface, ABI exports, bus tests
+    │   └── test_support.rs   # shared unit-test fixtures (cfg(test) only)
     ├── tests/          # integration tests against the public API only
     └── examples/       # runnable, compiled-in-CI usage examples
 vendor/tinybus/         # pinned TinyBus host types and module SDK
@@ -70,15 +49,38 @@ docs/
 └── adr/                # immutable architecture decision records
 ```
 
+### Configuration lives in JSON, not Rust
+
+Queue names and queue specs (attempts, dedupe windows, leases) are declared
+only in `queues.json`, a `tinystoragedrivers` `QueueManifest` the pipeline
+applies at construction. The pipeline's tunables are in `pipeline.json`,
+which refers to queues by name. Both are parsed strictly: an unknown field is
+an error, and every error names the offending field's JSON path. When you add
+a tunable, add it to the contract's `PipelineConfig` (with
+`deny_unknown_fields`), to `schemas/pipeline.schema.json`, to the shipped
+`pipeline.json`, and a range check with a field-path error in the module's
+`config` module — in the same commit. The tests check that the shipped files
+load, resolve against each other, and that the schema describes every field.
+
+### Storage
+
+The pipeline is generic over `Arc<dyn QueueStore>` and `Arc<dyn DocumentStore>`
+from `tinystoragedrivers-core`, plus an injectable `Clock`. Every piece of
+durable state lives in those two stores, so a new `Pipeline` over the same
+storage is a restart; tests simulate restarts exactly that way, on
+`MemoryStorage` with a `ManualClock`. `tinystoragedrivers-core` is a git
+dependency pinned to a tinyhumansai/tinystoragedrivers `rev`; move the pin in
+its own commit.
+
 ### The two-crate split
 
-`crates/template-bus` holds every type that crosses the bus and the names of the
+`crates/tinyeventqueues-bus` holds every type that crosses the bus and the names of the
 members that carry them. It has no transport, no runtime, and no behavior, and
 CI asserts it stays that way. A host that only makes calls depends on it alone.
 
-`crates/template` depends on it and re-exports all of it, so
-`template::GreetRequest` and `template_bus::GreetRequest` are the *same* type
-rather than structural twins. That direction is load-bearing: a parallel set of
+`crates/tinyeventqueues` depends on it and re-exports all of it, so
+`tinyeventqueues::InboundEvent` and `tinyeventqueues_bus::InboundEvent` are the
+*same* type rather than structural twins. That direction is load-bearing: a parallel set of
 payload types for hosts would mean a conversion at every call site that nothing
 checks.
 
@@ -99,22 +101,25 @@ workspace = true
 Each feature area belongs in a focused module directory under a crate's `src/`.
 A module root explains the module, wires its pieces together, and exposes the
 smallest useful API. Move substantial type definitions into `types.rs` and put
-module-local unit tests in a dedicated `test.rs`, wired from the bottom of the
-module root with:
+module-local unit tests in a sibling `mod_tests.rs` (or `<module>_tests.rs`
+beside a `<module>.rs`), wired from the bottom of the module root with:
 
 ```rust
 #[cfg(test)]
-mod test;
+#[path = "mod_tests.rs"]
+mod tests;
 ```
 
-Do not accumulate inline `mod tests` blocks in implementation files, and do not
-let a general-purpose `utils.rs` or `helpers.rs` grow — those are a symptom of a
+Never name a test file `test.rs`, `tests.rs` or `<module>_test.rs`, and never
+write an inline `#[cfg(test)] mod tests { ... }` block: OpenHuman vendors this
+repository, and its layout check rejects all of those. This deviates from the
+upstream rust-template on purpose. Do not let a general-purpose `utils.rs` or `helpers.rs` grow — those are a symptom of a
 missing module. Prefer many small modules that each do one thing well over few
 broad ones.
 
 Keep public exports centralized in each crate's `src/lib.rs` so downstream users
 have one predictable surface. Put shared error variants in
-`crates/template/src/error/mod.rs` and return the crate-wide `Result<T>` from
+`crates/tinyeventqueues/src/error/mod.rs` and return the crate-wide `Result<T>` from
 fallible public APIs.
 
 ## Build And Test
@@ -133,8 +138,8 @@ Supporting commands:
 
 - `cargo fmt --all` — format before committing.
 - `cargo test <filter>` — run a focused subset while iterating.
-- `cargo test -p template-bus` — run one crate's suite.
-- `cargo run -p template --example basic` — run the bundled example.
+- `cargo test -p tinyeventqueues-bus` — run one crate's suite.
+- `cargo run -p tinyeventqueues --example basic` — run the bundled example.
 - `cargo doc --no-deps --all-features` — build the rustdoc CI also builds with
   `RUSTDOCFLAGS="-D warnings"`.
 - `cargo test --doc` — run doctests alone when editing documentation examples.
@@ -185,7 +190,7 @@ add one:
 - gate anything optional behind a Cargo feature, documented in `Cargo.toml`;
 - declare it once in the root `[workspace.dependencies]` when more than one
   crate needs it, and take it with `{ workspace = true }`;
-- never add one to `crates/template-bus` that pulls in a transport, an async
+- never add one to `crates/tinyeventqueues-bus` that pulls in a transport, an async
   runtime, an HTTP client, or a native library — CI fails the build if you do;
 - leave a comment above the entry explaining *why* the crate is needed and what
   uses it — see the existing entries for the expected tone;
@@ -211,8 +216,9 @@ new module capability requires more.
 
 ## Testing
 
-- Module-local unit tests live in `crates/<crate>/src/<feature>/test.rs` and may
-  touch private items.
+- Module-local unit tests live in `crates/<crate>/src/<feature>/mod_tests.rs`
+  and may touch private items; shared fixtures are in
+  `crates/tinyeventqueues/src/test_support.rs`.
 - Integration tests live in `crates/<crate>/tests/` and exercise only the public
   API — they are the regression suite for the crate's contract.
 - Payload types pin their serde representation in a unit test. That
@@ -240,7 +246,7 @@ Write documentation for the reader who has never seen the code.
 
 - Every public item gets a rustdoc comment. `missing_docs` is a warning that CI
   treats as an error.
-- Start every `mod.rs` and `test.rs` with a concise module-level `//!`
+- Start every `mod.rs` and `mod_tests.rs` with a concise module-level `//!`
   description.
 - Each crate's `src/lib.rs` carries its crate-level overview: what the crate
   does, the primary entry points, and a short runnable example. It should also
@@ -295,7 +301,7 @@ Releases run from `.github/workflows/release.yml` via a manual
 an interrupted release after its version commit and tag exist. The workflow
 re-runs the full validation suite, computes the next version, updates
 the root `[workspace.package]` version and `Cargo.lock`, commits and tags
-`vX.Y.Z`, builds `crates/template` as a TinyBus module for every supported
+`vX.Y.Z`, builds `crates/tinyeventqueues` as a TinyBus module for every supported
 platform, pushes, and creates an immutable GitHub release with installable
 native packages.