Repository navigation
feat(queue): QueueStore port with a DocumentQueue over any document store - #71
Conversation
…cument store Durable at-least-once queues: leases with a visibility window, attempt counting, dead letters and redrive, priorities, delays, per-group serialization and dedupe keys with an acknowledgement window. DocumentQueue implements every feature over DocumentStore::claim and version-guarded writes, so memory, SQLite, file and MongoDB are queue backends without driver code. The queue checks run in run_ports. Co-authored-by: Medulla <medulla@tinyhumans.ai>
Tiny Sweeper reviewTiny Sweeper reviewed this change across 5 lane(s) and found 0 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below. State: Incomplete Review snapshot
Completeness: Incomplete FeaturesNone identified with supported citations. TestsNo supported feature-to-test mapping was produced. Test execution is not inferred.
FindingsNo active actionable findings. Could not review: README.md, crates/tinystoragedrivers-core/src/queue/document.rs, crates/tinystoragedrivers-core/src/queue/manifest.rs, crates/tinystoragedrivers-core/src/queue/manifest_tests.rs, crates/tinystoragedrivers-core/src/queue/mod.rs, docs/examples/queues.json, docs/plans/queue-store.md, docs/specs/queue-store.md, schemas/queues.schema.json, tinysweeper/description, tinysweeper/tests Before merge
How this fits togetherflowchart LR
n0["enqueue"]:::impacted
n1["attempts_and_dead_letters"]:::impacted
n2["dedupe"]:::impacted
n3["delays_and_nack"]:::impacted
n4["extend"]:::impacted
n5["groups"]:::impacted
n1 -->|calls| n0
n2 -->|calls| n0
n3 -->|calls| n0
n4 -->|calls| n0
n5 -->|calls| n0
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Agent review detailscritique
security
tests
commits
description
Evidence and run details
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
This review includes 17 billable files and costs up to $4.25. Or wait 25 minutes for your next included review. View limit details
Comment |
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
There was a problem hiding this comment.
tinysweeper found nothing blocking, but could not review everything, so this is not an approval: README.md, crates/tinystoragedrivers-core/src/conformance/mod.rs, crates/tinystoragedrivers-core/src/conformance/queue.rs, crates/tinystoragedrivers-core/src/lib.rs, crates/tinystoragedrivers-core/src/queue/document.rs, crates/tinystoragedrivers-core/src/queue/document_tests.rs, crates/tinystoragedrivers-core/src/queue/mod.rs, crates/tinystoragedrivers-core/src/queue/mod_tests.rs and 7 more.
$0.0001 · 0 in / 0 out · 780 embedded · ladder/vectors-oai3
QueueManifest loads queues.json (strict: unknown fields, duplicate or invalid names and bad values fail naming the entry) and apply() declares every queue on a QueueStore, so queues are added or tuned by editing a file instead of code. schemas/queues.schema.json documents the format for editors and agents, a test keeps it in step with QueueDeclaration, and docs/examples/queues.json is a working example. Co-authored-by: Medulla <medulla@tinyhumans.ai>
There was a problem hiding this comment.
tinysweeper found nothing blocking, but could not review everything, so this is not an approval: README.md, crates/tinystoragedrivers-core/src/queue/document.rs, crates/tinystoragedrivers-core/src/queue/manifest.rs, crates/tinystoragedrivers-core/src/queue/manifest_tests.rs, crates/tinystoragedrivers-core/src/queue/mod.rs, docs/examples/queues.json, docs/plans/queue-store.md, docs/specs/queue-store.md and 3 more.
$0.0001 · 0 in / 0 out · 786 embedded · ladder/vectors-oai3
1 participant
Summary
Adds a fourth port,
QueueStore: durable at-least-once message queues. Consumers lease messages for a visibility window, then ack, nack (retry after a delay, or dead-letter) or extend the lease. A lapsed lease means another delivery with the attempt count raised. A message that runs out of attempts is dead-lettered, and a dead letter can be redriven. Optional semantics are reported inQueueFeatures: priorities, delays, groups (at most one leased message per group, so one conversation is serialized without serializing the whole queue) and dedupe keys (a redelivery is rejected while the message is held and for a window after it is acked).Queues are declared in an editable JSON manifest instead of in code.
QueueManifest::load("queues.json")followed byapply(&store)declares every queue the file lists. The format is inschemas/queues.schema.json, so people and other agents can add or tune a queue by editing one file. Parsing is strict: a typo, a duplicate or an invalid value fails with an error that names the entry.queue::DocumentQueueimplements all of it overDocumentStore::claimand version-guarded writes, so memory, SQLite, file and MongoDB are queue backends without driver code. It is the first step towardstinyhumansai/tinyeventqueues, a module that buffers and filters noisy inbound events (chat messages, webhooks, connector triggers) before they reach an agent. A native RabbitMQ driver and the storage-bus selectors follow in separate PRs, as listed indocs/plans/queue-store.md.Related issue
None.
API or behavior changes
Additive only. New public module
queue, withQueueStorere-exported at the crate root. New:QueueManifest/QueueDeclaration,schemas/queues.schema.jsonanddocs/examples/queues.json. New reserved collections:tsd.queue,tsd.queue_groupsandtsd.queue_specs.conformance::run_portsnow also runs the queue checks, so every driver and the bus-client loopback exercise them. No existing behavior changes.Validation
cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningscargo build --all-targets --all-featurescargo test --all-features: the memory, SQLite, file and client-loopback suites all pass.TSD_MONGO_URL, against a throwaway single-node replica set from.github/scripts/start-mongo-replset.sh): passes bothlive_conformance_on_a_replica_setandlive_conformance_without_transactions.cargo llvm-cov -p tinystoragedrivers-core: line coverage is 96.5% forqueue/document.rs98.6% forqueue/manifest.rs, and 100% forqueue/types.rsandconformance/queue.rs.Tests
conformance/queue.rsdrives the queue with a hand-moved clock, so leases, delays and dedupe windows are tested without sleeping. It covers:Manifest tests check that the example loads, defaults, the errors for typos, bad values and duplicates (each naming the entry), file loading (missing file, directory, invalid content), that
applypersists specs, and that the JSON Schema lists exactly theQueueDeclarationfields. Unit tests cover name and message validation, sequence monotonicity when the clock steps back, and spec caching.Two behaviors are documented but not asserted:
Documentation
docs/specs/queue-store.md(the contract)docs/plans/queue-store.md(the delivery plan, with remaining tasks)Checklist
#[allow(...)],#[ignore], or relaxed lints. There are two scoped allows, each with a reason:clippy::expect_usedonDocumentQueue::new, for the constant collection names.Leases::newuses the same pattern.clippy::struct_excessive_boolsonQueueFeatures, whose fields are independent feature flags..envcontents in the diff or the description