Repository navigation
Dispatch with retries and dead letters, no-loss and flood tests - #5
Conversation
…ent contract Co-authored-by: Medulla <medulla@tinyhumans.ai>
…er the storage ports Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
… a digest Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
…ueue port Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Co-authored-by: Medulla <medulla@tinyhumans.ai>
…l the host over the bus Co-authored-by: Medulla <medulla@tinyhumans.ai>
Tiny Sweeper reviewTiny Sweeper reviewed this change across 5 lane(s) and found 0 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below. State: Incomplete Review snapshot
Completeness: Incomplete FeaturesNone identified with supported citations. TestsNo supported feature-to-test mapping was produced. Test execution is not inferred.
FindingsNo active actionable findings. Could not review: .github/ISSUE_TEMPLATE/config.yml, .github/workflows/ci.yml, .github/workflows/release.yml, AGENTS.md, Cargo.toml, MODULE.md, README.md, ROADMAP.md, crates/tinyeventqueues-bus/Cargo.toml, crates/tinyeventqueues-bus/README.md, crates/tinyeventqueues-bus/src/config/mod.rs, crates/tinyeventqueues-bus/src/config/mod_tests.rs, crates/tinyeventqueues-bus/src/config/types.rs, crates/tinyeventqueues-bus/src/event/mod.rs, crates/tinyeventqueues-bus/src/event/mod_tests.rs, crates/tinyeventqueues-bus/src/event/types.rs, crates/tinyeventqueues-bus/src/host/mod.rs, crates/tinyeventqueues-bus/src/host/mod_tests.rs, crates/tinyeventqueues-bus/src/host/types.rs, crates/tinyeventqueues-bus/src/lib.rs, crates/tinyeventqueues-bus/src/names/mod.rs, crates/tinyeventqueues-bus/src/names/mod_tests.rs, crates/tinyeventqueues-bus/src/ops/mod.rs, crates/tinyeventqueues-bus/src/ops/mod_tests.rs, crates/tinyeventqueues-bus/src/ops/types.rs, crates/tinyeventqueues-bus/src/outcome/mod.rs, crates/tinyeventqueues-bus/src/outcome/mod_tests.rs, crates/tinyeventqueues-bus/src/outcome/types.rs, crates/tinyeventqueues-bus/src/version/mod.rs, crates/tinyeventqueues-bus/src/version/mod_tests.rs, crates/tinyeventqueues/Cargo.toml, crates/tinyeventqueues/examples/basic.rs, crates/tinyeventqueues/examples/verify_github_release.rs, crates/tinyeventqueues/examples/verify_module.rs, crates/tinyeventqueues/src/cadence/mod.rs, crates/tinyeventqueues/src/cadence/mod_tests.rs, crates/tinyeventqueues/src/cadence/spec.rs, crates/tinyeventqueues/src/cadence/spec_tests.rs, crates/tinyeventqueues/src/clock/mod.rs, crates/tinyeventqueues/src/clock/mod_tests.rs, crates/tinyeventqueues/src/coalesce/mod.rs, crates/tinyeventqueues/src/coalesce/mod_tests.rs, crates/tinyeventqueues/src/coalesce/types.rs, crates/tinyeventqueues/src/config/mod.rs, crates/tinyeventqueues/src/config/mod_tests.rs, crates/tinyeventqueues/src/config/types.rs, crates/tinyeventqueues/src/dead/mod.rs, crates/tinyeventqueues/src/dead/mod_tests.rs, crates/tinyeventqueues/src/dead/types.rs, crates/tinyeventqueues/src/digest/mod.rs, crates/tinyeventqueues/src/digest/mod_tests.rs, crates/tinyeventqueues/src/error/mod.rs, crates/tinyeventqueues/src/error/mod_tests.rs, crates/tinyeventqueues/src/gate/ambient.rs, crates/tinyeventqueues/src/gate/buckets.rs, crates/tinyeventqueues/src/gate/mod.rs, crates/tinyeventqueues/src/gate/mod_tests.rs, crates/tinyeventqueues/src/host/mod.rs, crates/tinyeventqueues/src/ingress/mod.rs, crates/tinyeventqueues/src/ingress/mod_tests.rs, crates/tinyeventqueues/src/ingress/types.rs, crates/tinyeventqueues/src/lib.rs, crates/tinyeventqueues/src/observe/mod.rs, crates/tinyeventqueues/src/observe/mod_tests.rs, crates/tinyeventqueues/src/pipeline/builder.rs, crates/tinyeventqueues/src/pipeline/cadence_tests.rs, crates/tinyeventqueues/src/pipeline/dispatch.rs, crates/tinyeventqueues/src/pipeline/dispatch_tests.rs, crates/tinyeventqueues/src/pipeline/invariants_tests.rs, crates/tinyeventqueues/src/pipeline/mod.rs, crates/tinyeventqueues/src/pipeline/mod_tests.rs, crates/tinyeventqueues/src/pipeline/pass.rs, crates/tinyeventqueues/src/pipeline/pass_tests.rs, crates/tinyeventqueues/src/salience/mod.rs, crates/tinyeventqueues/src/salience/mod_tests.rs, crates/tinyeventqueues/src/salience/simhash.rs, crates/tinyeventqueues/src/store/ledger.rs, crates/tinyeventqueues/src/store/mod.rs, crates/tinyeventqueues/src/store/mod_tests.rs, crates/tinyeventqueues/src/test_support.rs, crates/tinyeventqueues/src/tinybus_module/README.md, crates/tinyeventqueues/src/tinybus_module/mod.rs, crates/tinyeventqueues/src/tinybus_module/mod_tests.rs, crates/tinyeventqueues/src/turn/mod.rs, crates/tinyeventqueues/src/turn/mod_tests.rs, crates/tinyeventqueues/src/turn/threads.rs, crates/tinyeventqueues/src/turn/threads_tests.rs, crates/tinyeventqueues/tests/public_api.rs, deny.toml, docs/README.md, docs/plans/README.md, docs/plans/event-pipeline.md, docs/plans/tinybus-module-release.md, docs/specs/README.md, docs/specs/event-pipeline.md, docs/specs/tinybus-module-release.md, pipeline.json, queues.json, schemas/pipeline.schema.json, tinysweeper/description, tinysweeper/tests Before merge
Agent review detailscritique
security
tests
commits
description
Evidence and run details
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
tinysweeper found nothing blocking, but could not review everything, so this is not an approval: .github/ISSUE_TEMPLATE/config.yml, .github/workflows/ci.yml, .github/workflows/release.yml, AGENTS.md, Cargo.toml, MODULE.md, README.md, ROADMAP.md and 93 more.
$0.0001 · 0 in / 0 out · 751 embedded · ladder/vectors-oai3
Summary
Last of five stacked PRs for the inbound event pipeline. Stacked on #4 (and #3, #2, #1): this PR's own commits are "Dispatch turns to the host with retries, backoff and dead letters" and "Keep thread order across lanes, test no-loss and a 10k flood, and call the host over the bus" (plus a merge of the re-pinned #4).
pipeline/dispatch.rs): eachDrainpass visits the lanes most urgent first and hands up to each lane'sconcurrencyturns to the host'sDeliver.accepted→ events enddelivered(digeston the digest lane);retryor a failed call → nack with exponential backoff (dispatch.backoff_base_ms..backoff_max_ms, or the host'safter_ms); out of the lane queue's manifestmax_attempts→ dead-lettered withmax_attempts;rejected→ dead-lettered at once. Dead turns are parked in the dead-letter queue (ListDead/Redrive).Turn.platform_ackslists events ofafter_dispatch/after_turnsources.teq.threads) sends a new turn to the most urgent lane its thread has queued work on and folds queued turns of slower lanes into it (those are acknowledged unseen when reached). Staged cadence entries are now one per thread, and interactive work releases its thread's waiting entry whatever the cadence.dispatch.gap_timeout_ms, but never while that turn is backing off; a redelivered finished turn is acknowledged only if all its events ended, otherwise delivered rather than lost.BusHostthat calls the host'sai.tinyhumans.tinyeventqueues.Host(ClassifySalience,Deliver) over the bus; the module keeps exporting all six members.pipeline.jsongainsdispatch(schema, field-path validation).Related issue
None. Builds on tinyhumansai/tinystoragedrivers#71-#73 (merged):
tinystoragedrivers-coreis pinned togit = "https://github.com/tinyhumansai/tinystoragedrivers", rev = "0c6dcfce57201fadd830431f36fd759c3baa897a".API or behavior changes
Additive:
DispatchConfig,Host::deliver(theHosttrait gains a required method, unreleased),DrainReport.dispatched/deadnow filled. Behavior: a pipeline with a host now dispatches duringDrain.Validation
cargo fmt --all -- --check— cleancargo clippy --all-targets --all-features -- -D warnings— cleancargo build --all-targets --all-features— ok;cargo +1.88.0 build --all-targets --all-featuresokcargo test --all-features/cargo test— 134 + 5 + 35 tests, 4 doctests pass (~20 s, mostly the flood test)RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features— cleancargo run -p tinyeventqueues --example basic— ok.github/scripts/check-file-coverage.sh 90 coverage.json— every file ≥ 90% (lowest 93.75%)Tests
pipeline/invariants_tests.rs): randomized mixed traffic (six sources, edits/deletes/replies targeting earlier events, bots, self-echoes, media, redeliveries, invalid envelopes) through a host that randomly accepts, retries, rejects and fails, on a DocumentQueue and on the broker double (no dedupe, no groups). After quiescing, every accepted event has exactly one terminal outcome in the ledger, the sink saw exactly one terminal outcome per event, totals match, and every queue is empty.max_attemptsthen redrive and delivery, rejection, deferred platform acks, unreadable turns, conversation order without groups under retries, gap-timeout skipping and redelivery handling; the thread registry and turnstile unit tests; a public-API host test; host callbacks over a real in-memory TinyBus.Remaining gaps
tinystoragedriversstorage bus is a follow-up. Passes run when the host callsDrain(no module-side timer yet).concurrencybounds turns per lane per pass (deliveries within a pass are sequential).mainalready (RUSTSEC advisories forrustls/zipvia the vendored TinyBus).Documentation
README, ROADMAP, MODULE, spec status, plan (design decisions), adapter README,
pipeline.jsonand its schema.Checklist
#[allow(...)],#[ignore], or relaxed lints.envcontents in the diff or the description