Update project dependencies and E2E coverage#82
Merged
Conversation
Remove the VS Code workspace file from repository tracking while leaving local workspace files ignored.
Simplify the just test recipe to run cargo test directly. Remove the obsolete test-output cache ignore entry and update contributor docs.
The _FOR_USER variants supersede it; the constant had no remaining references.
The column in object_storage_deletions (V8) and email_outbox (V12) was written in the claim and cleared in mark-failed but never read; reclaim is gated solely on next_attempt_at. Remove it from both migrations (edited in place, pre-merge) and from the two CLAIM and two MARK_FAILED queries.
The presign-batch cap was defined twice (backend usize, frontend i64) and must agree. Move it to the shared crate (alongside ALLOWED_MIME_TYPES) as the single source of truth; backend imports it, frontend casts to i64 at the use site.
Add worker::MaintenanceTask (run_once as RPITIT returning impl Future + Send, so the tokio::spawn driver can require Send) and worker::spawn, which owns the tick loop and missed-tick-delay behaviour. ObjectLifecycleWorker and EmailWorker now implement the trait instead of each carrying duplicate spawn/run_forever loops; worker_interval drops unsigned_abs for a plain cast. main.rs calls worker::spawn.
…nce interval The three outboxes share lease/retry validation but have genuinely different defaults (batch_size 1000 for storage deletion to match S3's multi-delete cap, 100 for email), so a shared serde struct cannot express them. Add an OutboxRetryConfig runtime view (not deserialized) built by per-owner accessors (storage_deletion/upload_session_cleanup/retry), plus an ensure_positive! macro; the configs keep their flat fields and validate through the view, collapsing the duplicated > 0 checks. Rename storage_deletion_worker_interval_seconds to maintenance_interval_seconds (it drives the whole maintenance pass), including the env key in .env.example, docs, and README.
Config::from_env becomes Config::load: when MEMORY_MAP_CONFIG is set it reads that TOML file (required) layered under the environment source, so env still wins and secrets can stay in a gitignored config.toml; when unset, loading is pure-environment and byte-for-byte identical to before. Add config.example.toml (flat outbox keys, blank secrets), an opt-in 'just config' recipe, .gitignore entries, the layered model in docs/deployment.md, and a test asserting the example deserializes into Config so it cannot drift from the structs.
Extract OutboxQueue/OutboxProcessor traits, DrainOutcome, and a generic drain_outbox loop into outbox.rs, replacing the two near-identical drain_storage_deletion_outbox/drain_email_outbox functions. Storage and email each get a newtype queue wrapper and a processor (storage deletes a whole claim in one S3 request and is all-or-nothing; email sends per message and isolates failures), reusing the existing SQL consts unchanged. EmailSender keeps its test seam but moves to RPITIT + Send so the generic processor future stays Send; StorageDeletionSink is dropped (the storage path is integration-covered). The per-queue drain unit tests collapse into one generic pair plus an email processor test.
…Config view claim_expired_upload_sessions and mark_upload_session_cleanup_failed now read their lease/retry knobs from config.upload_session_cleanup() instead of the flat fields, matching how the storage and email outboxes read theirs. The reconcile stays bespoke (not drain_outbox): it reconciles a primary record and fans out into the deletion queue, and its claim joins objects and filters storage_state. A comment records that rationale.
The 9-column projection that S3Object::try_from decodes by name was copy-pasted into five transitional-state mutation queries (INSERT_OBJECT, UPDATE_OBJECT, and the three delete-pending marks). Extract an object_returning_columns! macro and compose each const via concat!, so a column or alias change touches one place and cannot silently drift from the struct's by-name reads.
…accessor Add ContextWrapper::object_lifecycle_service, which borrows the request's storage client and lifecycle config over a caller-provided db client. The six object-lifecycle mutations now build the service through it, dropping the repeated storage/state lookups and config clone from each resolver.
Share the part-count and part-length arithmetic behind pure part_count/part_length helpers (the two callers keep their own intentionally different part-count caps); inline the s3_object_from_row pass-through into S3Object::try_from; and log the maintenance success counts at info rather than warn (the error branches stay warn).
Both storage methods now have no production callers (the presigned multipart upload path replaced direct upload, and metadata is read via head_object). Add a doc comment on each recording that they are exercised only by the integration tests and kept as public storage primitives / test seams; gating them behind cfg(test) would break the integration tests, which compile against the crate built without cfg(test).
Force Content-Disposition: attachment on presigned GET URLs for script-capable content types (currently image/svg+xml) so a direct top-level navigation downloads the file instead of executing embedded script in the storage origin. In-app img/video embedding ignores the header, so display is unaffected; the guard is per content type to keep safe media viewable on navigation. Thread an optional content disposition through presigned_get_url and cover the round-trip in the storage integration test.
- Split object_lifecycle.rs into config/service/worker/deletion submodules using the project's parent-file convention, preserving the public API via re-exports. - Single-source the upload-session column projection with a macro and trim the four queries to the columns actually decoded. - Redact object name, timestamp, and coordinates from the update-metadata debug log, keeping the id and a has_location flag. - Surface a parked-backlog warning each maintenance pass for cleanups and deletions that exhausted their retry budget, backed by two count queries. - Document the password-reset outbox plaintext-token trade-off, and add the incomplete-multipart lifecycle and storage-origin-isolation notes to the deployment guide.
The workflow granted id-token: write for the FlakeHub cache, which was then disabled in favor of the GitHub Actions cache (authenticated by the runtime token, not OIDC). Remove the unused permission and keep contents: read.
Rename the previous service-skipping verify recipe to verify-fast and make verify run it plus the service-backed storage, backend-integration, and e2e suites serially. CI keeps invoking the recipes as parallel jobs. Add verify-fast to the filtered allowlist and refresh the contributor docs, including the RustFS rename and the project-structure listing.
Add a counterpart to the config.example.toml deserialization test that feeds the .env.example MEMORY_MAP__ pairs through the same environment source Config::load uses, so a renamed or removed key fails CI instead of drifting silently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification