Repository navigation
Cadence scheduler with ingestion budgets and overflow folding - #4
Conversation
…ent contract Co-authored-by: Medulla <medulla@tinyhumans.ai>
…er the storage ports Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
… a digest Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Tiny Sweeper reviewTiny Sweeper reviewed this change across 5 lane(s) and found 0 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below. State: Incomplete Review snapshot
Completeness: Incomplete FeaturesNone identified with supported citations. TestsNo supported feature-to-test mapping was produced. Test execution is not inferred.
FindingsNo active actionable findings. Could not review: .github/ISSUE_TEMPLATE/config.yml, .github/workflows/ci.yml, .github/workflows/release.yml, AGENTS.md, Cargo.toml, MODULE.md, README.md, ROADMAP.md, crates/tinyeventqueues-bus/Cargo.toml, crates/tinyeventqueues-bus/README.md, crates/tinyeventqueues-bus/src/config/mod.rs, crates/tinyeventqueues-bus/src/config/mod_tests.rs, crates/tinyeventqueues-bus/src/config/types.rs, crates/tinyeventqueues-bus/src/event/mod.rs, crates/tinyeventqueues-bus/src/event/mod_tests.rs, crates/tinyeventqueues-bus/src/event/types.rs, crates/tinyeventqueues-bus/src/host/mod.rs, crates/tinyeventqueues-bus/src/host/mod_tests.rs, crates/tinyeventqueues-bus/src/host/types.rs, crates/tinyeventqueues-bus/src/lib.rs, crates/tinyeventqueues-bus/src/names/mod.rs, crates/tinyeventqueues-bus/src/names/mod_tests.rs, crates/tinyeventqueues-bus/src/ops/mod.rs, crates/tinyeventqueues-bus/src/ops/mod_tests.rs, crates/tinyeventqueues-bus/src/ops/types.rs, crates/tinyeventqueues-bus/src/outcome/mod.rs, crates/tinyeventqueues-bus/src/outcome/mod_tests.rs, crates/tinyeventqueues-bus/src/outcome/types.rs, crates/tinyeventqueues-bus/src/version/mod.rs, crates/tinyeventqueues-bus/src/version/mod_tests.rs, crates/tinyeventqueues/Cargo.toml, crates/tinyeventqueues/examples/basic.rs, crates/tinyeventqueues/examples/verify_github_release.rs, crates/tinyeventqueues/examples/verify_module.rs, crates/tinyeventqueues/src/cadence/mod.rs, crates/tinyeventqueues/src/cadence/mod_tests.rs, crates/tinyeventqueues/src/cadence/spec.rs, crates/tinyeventqueues/src/cadence/spec_tests.rs, crates/tinyeventqueues/src/clock/mod.rs, crates/tinyeventqueues/src/clock/mod_tests.rs, crates/tinyeventqueues/src/coalesce/mod.rs, crates/tinyeventqueues/src/coalesce/mod_tests.rs, crates/tinyeventqueues/src/coalesce/types.rs, crates/tinyeventqueues/src/config/mod.rs, crates/tinyeventqueues/src/config/mod_tests.rs, crates/tinyeventqueues/src/config/types.rs, crates/tinyeventqueues/src/dead/mod.rs, crates/tinyeventqueues/src/dead/mod_tests.rs, crates/tinyeventqueues/src/dead/types.rs, crates/tinyeventqueues/src/digest/mod.rs, crates/tinyeventqueues/src/digest/mod_tests.rs, crates/tinyeventqueues/src/error/mod.rs, crates/tinyeventqueues/src/error/mod_tests.rs, crates/tinyeventqueues/src/gate/ambient.rs, crates/tinyeventqueues/src/gate/buckets.rs, crates/tinyeventqueues/src/gate/mod.rs, crates/tinyeventqueues/src/gate/mod_tests.rs, crates/tinyeventqueues/src/host/mod.rs, crates/tinyeventqueues/src/ingress/mod.rs, crates/tinyeventqueues/src/ingress/mod_tests.rs, crates/tinyeventqueues/src/ingress/types.rs, crates/tinyeventqueues/src/lib.rs, crates/tinyeventqueues/src/observe/mod.rs, crates/tinyeventqueues/src/observe/mod_tests.rs, crates/tinyeventqueues/src/pipeline/builder.rs, crates/tinyeventqueues/src/pipeline/cadence_tests.rs, crates/tinyeventqueues/src/pipeline/mod.rs, crates/tinyeventqueues/src/pipeline/mod_tests.rs, crates/tinyeventqueues/src/pipeline/pass.rs, crates/tinyeventqueues/src/pipeline/pass_tests.rs, crates/tinyeventqueues/src/salience/mod.rs, crates/tinyeventqueues/src/salience/mod_tests.rs, crates/tinyeventqueues/src/salience/simhash.rs, crates/tinyeventqueues/src/store/ledger.rs, crates/tinyeventqueues/src/store/mod.rs, crates/tinyeventqueues/src/store/mod_tests.rs, crates/tinyeventqueues/src/test_support.rs, crates/tinyeventqueues/src/tinybus_module/README.md, crates/tinyeventqueues/src/tinybus_module/mod.rs, crates/tinyeventqueues/src/tinybus_module/mod_tests.rs, crates/tinyeventqueues/src/turn/mod.rs, crates/tinyeventqueues/src/turn/mod_tests.rs, crates/tinyeventqueues/tests/public_api.rs, deny.toml, docs/README.md, docs/plans/README.md, docs/plans/event-pipeline.md, docs/plans/tinybus-module-release.md, docs/specs/README.md, docs/specs/event-pipeline.md, docs/specs/tinybus-module-release.md, pipeline.json, queues.json, schemas/pipeline.schema.json, tinysweeper/description, tinysweeper/tests Before merge
Agent review detailscritique
security
tests
commits
description
Evidence and run details
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
tinysweeper found nothing blocking, but could not review everything, so this is not an approval: .github/ISSUE_TEMPLATE/config.yml, .github/workflows/ci.yml, .github/workflows/release.yml, AGENTS.md, Cargo.toml, MODULE.md, README.md, ROADMAP.md and 88 more.
$0.0001 · 0 in / 0 out · 759 embedded · ladder/vectors-oai3
|
Warning Review limit reached
This review includes 114 billable files and costs up to $28.50. Or wait 16 minutes for your next included review. View limit details
Comment |
…ueue port Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
There was a problem hiding this comment.
tinysweeper found nothing blocking, but could not review everything, so this is not an approval: AGENTS.md, Cargo.toml, README.md, ROADMAP.md, crates/tinyeventqueues/Cargo.toml, crates/tinyeventqueues/src/ingress/mod_tests.rs, deny.toml, docs/plans/event-pipeline.md and 3 more.
$0.0001 · 0 in / 0 out · 758 embedded · ladder/vectors-oai3
There was a problem hiding this comment.
tinysweeper found nothing blocking, but could not review everything, so this is not an approval: .github/ISSUE_TEMPLATE/config.yml, .github/workflows/ci.yml, .github/workflows/release.yml, AGENTS.md, Cargo.toml, MODULE.md, README.md, ROADMAP.md and 88 more.
$0.0001 · 0 in / 0 out · 758 embedded · ladder/vectors-oai3
Summary
Fourth of five stacked PRs for the inbound event pipeline. Stacked on #3 (and #2, #1): this PR's own commits are the last two ("Add cadence and budget configuration to the contract", "Schedule turns by cadence with ingestion budgets and overflow folding").
Routed turns now go through a cadence scheduler (
cadence/) before they reach a lane queue.sources[].cadence, wins) or per lane (lanes.<lane>.cadence):instant(default),every(<n><ms|s|m|h|d>)(ticks aligned to the epoch) orcron("<min> <hour> <day> <month> <weekday>")(UTC;*, numbers, ranges, steps, lists; standard day-of-month/day-of-week OR rule). Parsed strictly; a bad string is a config error naming its field.teq.staged, one entry per cadence and conversation thread, so a conversation's pending items fold into one entry. A tick releases at mostbudget.max_turns, oldest first; the rest wait for the next tick (deferred/over_budget), and waiting entries beyondmax_turns * backlog_ticksfold into the digest.teq.cadence(next tick, the firing tick, and how many turns it released). A release is counted before it is enqueued, so a crash mid-tick resumes the same tick with what is left of its budget: no skip, no double fire. Ticks missed while down collapse into one. A changed schedule restarts the ticks.degrade_debounce_ms(folding each thread), released within the budget when due; once tokens are available the source is instant again.pipeline.json: background laneevery(10m)with 20 turns per tick;composio:*andwebhook:*instant with per-second budgets. Schema updated.Also in the stack since the last review (pushed to #1-#3): the pipeline branches on
QueueStore::features(): without dedupe or groups (the RabbitMQ driver) it never setsdedupe_key/group, keeps tombstones (ingress and turn ids) in the DocumentStore, and aBrokertest double that reports both false and refuses messages that set them covers those paths.Related issue
None. Builds on tinyhumansai/tinystoragedrivers#71-#73 (merged):
tinystoragedrivers-coreis pinned togit = "https://github.com/tinyhumansai/tinystoragedrivers", rev = "0c6dcfce57201fadd830431f36fd759c3baa897a"(the main revision that merged the queue port, tinyhumansai/tinystoragedrivers#71-#73).API or behavior changes
Additive:
Budget,cadence/budgetonLaneConfigandSourcePolicy, the publiccadencemodule (Cadence,Cron).Drainreportsscheduled.Validation
cargo fmt --all -- --check— cleancargo clippy --all-targets --all-features -- -D warnings— cleancargo build --all-targets --all-features— ok;cargo +1.88.0 build --all-targets --all-featuresokcargo test --all-features— 113 + 4 + 34 tests, 4 doctests passRUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features— clean.github/scripts/check-file-coverage.sh 90 coverage.json— every file ≥ 90%Tests
Fake-clock tests (
pipeline/cadence_tests.rs,cadence/*_tests.rs):every(10m)source flooded with 1,000 events over 20 threads releases exactly the budget (5) per tick, each turn carrying its thread's 50 folded events, all 1,000 delivered over four ticks, nothing after;Documentation
README, ROADMAP, spec status, plan (design decisions recorded there);
pipeline.jsonand its schema.Checklist
#[allow(...)],#[ignore], or relaxed lints.envcontents in the diff or the description