fix(dash-spv): don't preload filters below the actually stored range on filter-download start

[QuantumExplorer]

Issue being fixed or feature implemented

Fixes #892. FiltersManager::start_download treated stored_filters_tip (a tip watermark) as proof that filter storage is contiguous from scan_start. When only tip-region filters exist and the wallet's scan floor is far below (birth height « tip), the preload read never-populated segments: SegmentCache::get_items debug-aborts (permanent app crash-loop on relaunch, since auto-start resumes the same scan state) and release builds silently return sentinel filter data zipped against real headers.

What was done?

• FilterStorage gains filter_start_height() (the segment cache already tracked it, it was just never exposed) and clear_filters().
• start_download computes stored_covers_scan = stored_filters_tip > 0 && stored_start <= scan_start. When stored filters exist but don't reach down to scan_start, they are discarded (warn-logged) and the download restarts from scan_start — merely skipping the preload isn't enough: resuming from the stored tip would leave a permanent hole below the island that the in-order store loop can never fill, and keeping the island through a backfill risks two disjoint islands plus Segment::insert non-sentinel asserts. Preload and initial-batch mark_verified gate on stored_covers_scan.
• SegmentCache::get_items now returns a typed InvalidArgument for ranges beginning below the lowest stored height (replacing debug-abort / release-sentinel behavior for that case); SegmentCache::clear() added; parse_segment_id helper also fixes a latent slice-panic on short filenames in load_or_new.

How Has This Been Tested?

• test_start_download_discards_stored_filters_above_scan_start — the exact dash-spv: filter restart with scan_start far below stored tip loads never-populated segments — debug abort / release sentinel reads #892 shape (filters stored 900..=1000, scan_start 100); verified it fails against the old gate at exactly the bad preload.
• test_start_download_preloads_when_stored_filters_cover_scan_start — regression guard for the covers case (behavior unchanged).
• Storage-layer tests: filter_start_height + clear_filters round-trip incl. persist/reopen; get_items below-start typed error; segment-cache clear (persisted + unpersisted).
• cargo test -p dash-spv: 477 passed, 0 failed (the dashd_masternode binary requires DASHD_PATH, pre-existing environmental). New tests also pass in --release, where the debug assertions are compiled out. fmt/clippy/--all-features --tests/dash-spv-ffi --all-targets clean.

Breaking Changes

None. FilterStorage implementors gain two methods with default-free trait additions (in-repo implementors updated).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added filter storage controls to report the earliest available filter height and clear stored filters.
  • Cleared filter data can be persisted and replaced with a new range starting at an earlier height.

• Bug Fixes

  • Requests below the available filter range now return a clear validation error.
  • Filter synchronization now discards incomplete stored ranges and restarts from the correct scan height.
  • Existing complete filter ranges continue to preload and scan as expected.

[coderabbitai]

Warning

Review limit reached

@QuantumExplorer, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 46 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 40c44c23-6921-4e85-a2ce-4afdd6a1e688

📥 Commits

Reviewing files that changed from the base of the PR and between 23225a8 and 828f707.

📒 Files selected for processing (3)

• dash-spv/src/storage/filters.rs
• dash-spv/src/storage/segments.rs
• dash-spv/src/sync/filters/manager.rs

📝 Walkthrough

Walkthrough

Filter storage now reports its lowest stored height and supports clearing. Segment reads reject ranges below that height. Filter synchronization clears sparse, incompatible stored ranges and restarts downloads from scan_start, while preserving preload behavior for contiguous coverage.

Changes

Filter restart safety

Layer / File(s)	Summary
Segment cache validation and clearing dash-spv/src/storage/segments.rs	Segment filenames use centralized parsing, lower-bound reads return StorageError::InvalidArgument, and clear removes persisted and in-memory segments after persistence.
Filter storage clearing API dash-spv/src/storage/filters.rs, dash-spv/src/storage/mod.rs	Filter storage exposes its lowest height and clearing operation, with implementations delegated through PersistentFilterStorage and DiskStorageManager; tests cover persistence and rebuilding from a lower range.
Scan-start coverage and restart dash-spv/src/sync/filters/manager.rs	start_download clears stored filters that do not cover scan_start, resets stored progress, and only preloads or verifies covered ranges; regression tests cover sparse and dense storage.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant FiltersManager
  participant DiskStorageManager
  participant PersistentFilterStorage
  participant SegmentCache
  participant FilterPeer
  FiltersManager->>DiskStorageManager: read stored tip and start
  DiskStorageManager->>PersistentFilterStorage: delegate storage query
  PersistentFilterStorage->>SegmentCache: return stored range
  FiltersManager->>DiskStorageManager: clear filters when scan_start is uncovered
  DiskStorageManager->>PersistentFilterStorage: delegate clear
  FiltersManager->>FilterPeer: request filters from scan_start
  FilterPeer-->>FiltersManager: return filter batch

Loading

Possibly related PRs

• dashpay/rust-dashcore#790: Introduced the StorageError::InvalidArgument behavior used by the new lower-bound validation.
• dashpay/rust-dashcore#816: Also adjusts FiltersManager::start_download frontier and restart handling.
• dashpay/rust-dashcore#835: Overlaps with stored filter tip/start handling in the filter synchronization state machine.

Suggested labels: ready-for-review

Suggested reviewers: xdustinface, zocolini

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately describes the main fix: avoiding preload below the actual stored filter range.
Linked Issues check	✅ Passed	The changes address #892 by checking actual stored range, clearing sparse filters, and preventing invalid reads during download restart.
Out of Scope Changes check	✅ Passed	The extra storage helpers and parsing updates support the filter-range fix and do not appear unrelated to the PR goals.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/filter-preload-sparse-storage

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@dash-spv/src/storage/segments.rs`:
- Around line 137-149: Update parse_segment_id to validate the complete filename
rather than parsing a fixed four-byte slice: strip the exact segment prefix,
separator, and data-file extension, then parse the entire remaining ID component
as u32. Reject names with extra characters, missing separators, or malformed IDs
while preserving None for invalid filenames.
- Around line 458-480: Update Storage::clear to return StorageResult<()>,
propagate both read_dir failures and individual directory-entry errors, and
complete the entire scan before clearing or queuing any cache state. Then update
clear_filters and its callers to propagate the clear result while preserving the
existing reset behavior on success.

In `@dash-spv/src/sync/filters/manager.rs`:
- Around line 218-219: Update the stored-range check near stored_covers_scan to
use filter_start_height presence, rather than stored_filters_tip > 0, so a
stored range of 0..=0 is treated as non-empty and preloads height zero. Add a
restart unit test covering storage containing only filter height zero.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 12e47b48-9f16-4cd1-b92d-4a56ad7eab77

📥 Commits

Reviewing files that changed from the base of the PR and between 38c689d and 23225a8.

📒 Files selected for processing (4)

• dash-spv/src/storage/filters.rs
• dash-spv/src/storage/mod.rs
• dash-spv/src/storage/segments.rs
• dash-spv/src/sync/filters/manager.rs

[codecov]

Codecov Report

❌ Patch coverage is 97.73585% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.44%. Comparing base (38c689d) to head (828f707).

Files with missing lines	Patch %	Lines
dash-spv/src/storage/mod.rs	0.00%	4 Missing ⚠️
dash-spv/src/storage/segments.rs	98.27%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #893      +/-   ##
==========================================
+ Coverage   74.39%   74.44%   +0.05%     
==========================================
  Files         327      327              
  Lines       74779    75032     +253     
==========================================
+ Hits        55629    55855     +226     
- Misses      19150    19177      +27     

Flag	Coverage Δ
core	77.29% <ø> (ø)
ffi	50.31% <ø> (ø)
rpc	20.00% <ø> (ø)
spv	91.11% <97.73%> (-0.03%)	⬇️
wallet	74.41% <ø> (ø)

Files with missing lines	Coverage Δ
dash-spv/src/storage/filters.rs	100.00% <100.00%> (ø)
dash-spv/src/sync/filters/manager.rs	97.74% <100.00%> (+0.13%)	⬆️
dash-spv/src/storage/segments.rs	96.15% <98.27%> (+0.34%)	⬆️
dash-spv/src/storage/mod.rs	86.15% <0.00%> (-1.08%)	⬇️

... and 4 files with indirect coverage changes

[QuantumExplorer]

Follow-up with the full root-cause timeline from the device where this reproduced (per-session dash_spv logs, testnet) — including why a clean CLI sync + restarts can't reproduce it, and what the discard actually does in practice.

How the sparse island forms

The state is manufactured by a mid-life client re-initialization from a checkpoint, not by plain sync + restart:

session A  18:45  Starting filter header sync from 0 …
           18:46  Starting filter download (scan_start=0, download_start=0, stored_filters_tip=0, …)
           → filters stored densely from 0 upward; wallet scan progresses past 1.1M

session B  19:08  🚀 Starting sync from checkpoint at height 200000 instead of genesis (requested start height: 227121)
           19:08  Starting filter header sync from 200000 to 208000
           19:14  Starting filter download (scan_start=1514654, download_start=1514654, stored_filters_tip=0, …)
           → the re-init RESET filter storage (tip=0), but the re-download resumed from the wallet's
             current synced position (1514654), NOT the new 200000 floor
           → result: filters stored 1514654..=tip, while the client's floor is the 200000 checkpoint

sessions C+  scan_start=200000 (checkpoint floor), stored range 1514654..=2504884
             → pre-fix: preload of 200000..204999 reads never-populated segments → SIGABRT crash-loop

So the mysterious scan_start=200000 is the checkpoint height the client re-initialized from (requested start height: 227121 resolved down to the 200000 checkpoint). A fresh CLI sync never re-initializes from a checkpoint mid-life, which is why restart testing after a clean sync doesn't hit it.

Repro recipe: sync with filters stored from a low height → restart the client with a start-height request that initializes from a checkpoint below the wallet's current synced height → the filter re-download resumes from the wallet position, leaving the gap → restart once more → pre-fix crash (debug) / sentinel filter reads (release).

What the discard does in practice (convergence)

With this PR on the same corrupted device state:

21:32  WARN Stored filters Some(1514654)..=2504884 do not reach scan start 200000; discarding them and re-downloading
21:32  Starting filter download (scan_start=200000, download_start=200000, stored_filters_tip=0, …)
21:54  Starting filter download (scan_start=200000, download_start=203000, stored_filters_tip=202999, …)   ← dense from the floor, restart resumes
00:35  Starting filter download (scan_start=2504990, download_start=2504990, stored_filters_tip=2504989, …) ← scan completed to tip

One discard total. After it, storage is dense from the scan floor, every later restart resumes normally, and the wallet completed the backfill scan to tip. It is a one-time self-heal that restores the dense-storage invariant, not a recurring reset.

The remaining latent bug (separate issue)

The island-manufacturer itself — the checkpoint re-init resetting filter storage but resuming the re-download from the wallet's synced height instead of the new floor — is a distinct bug; filing it separately. This PR makes the client self-heal from any such island regardless of origin (that session's mid-scan kill shapes can produce the same state), which is worth having even once the manufacturer is fixed.

🤖 Generated with Claude Code

[QuantumExplorer]

Good challenge on the timeline — the T1/T2 distinction is correct, my earlier comment conflated the checkpoint floor with the scan floor. Here are the exact lines that were asked for, plus the T2 evidence. All from the same device, same logs.

T1 session (island forms), post-re-init and session end:

19:14:31  Starting filter download (scan_start=1514654, download_start=1514654, stored_filters_tip=0, target=1516000)
…filters download and scan to tip…
19:30:04  Committed batch 2504884-2504884, committed_height now 2504884

Session ends cleanly at committed 2,504,884 — consistent state, exactly as predicted: T1 alone gives stored 1514654..=tip with a scan floor ~1.51M, which would never crash.

T3 (first crashing session, 70 seconds later):

19:31:15  Starting filter download (scan_start=200000, download_start=2504885, stored_filters_tip=2504884, target=2504884)

So T2 is real and sits between those two lines: the wallet's committed height (2,504,884) did not survive the restart. With committed effectively 0 and birth 0, scan_start = max(birth, committed+1).max(header_start_height) collapses to the 200000 checkpoint floor. The persisted wallet row today carries syncedHeight = 2504884 — the value existed to persist — so the reversion is about when/whether it was durably written and re-applied, not about the number being unknown.

T2 is conditional, and the condition looks like abrupt termination. Post-fix restarts on the same device:

21:54:44  Starting filter download (scan_start=200000, download_start=203000, stored_filters_tip=202999, …)   ← restart after SIGABRT-era terminations: floor collapsed again
02:48:19  Starting filter download (scan_start=2505047, download_start=2505047, stored_filters_tip=2505046, …) ← restart after a long clean session: resumed correctly

Every floor-collapse restart in the log history followed an abrupt kill (the pre-fix crash-loop itself, or forced terminations); the one restart that followed a healthy long-running session resumed at committed+1. That points at the sync-height persistence cadence in the embedding app/platform-wallet layer (not dash-spv) — likely only stamped on some pulse or graceful path that abrupt kills miss. Nasty interaction: the pre-fix crash-loop guaranteed abrupt terminations, so T2 re-armed itself on every relaunch.

Refinement of #894: the island needs BOTH halves — T1 (checkpoint re-init resets filter storage, refill resumes from the wallet position instead of the floor) AND T2 (committed height regressing across an abrupt restart). I'll note this on #894; the T2 half belongs to the embedding-app layer and I'll chase it there.

On discard cost — conceded. This island spanned ~990k filters; the discard re-fetched ~2.3M, so ~40% of prior work was thrown away rather than kept. "Cheap" was wrong for this shape. The claims that stand: it's one-time per island (convergence shown above), it's correctness-first (the release-mode alternative is silently wrong filter matches over the gap), and a backfill-below-the-island variant would be a legitimate future optimization on top of the same detection.

🤖 Generated with Claude Code

[QuantumExplorer]

Bullet-point reproduction recipes, as requested. Two levels: the minimal one reproduces the crash in the CLI today; the second is the exact path our device took.

Minimal repro (CLI, no app needed — pre-#893 debug build)

The crash needs only one condition: stored filter range starting above the wallet scan floor. Any way of creating it works:

1. Start the client with a wallet whose birth/start height is near the tip (or start from a checkpoint) → filter sync stores filters from that height upward: stored = [HIGH..tip].
2. Stop the client.
3. Restart the same client/storage with the scan floor below HIGH — the natural way: import/add a wallet with a lower birth height (the thing multiwallet + feat(dash-spv): force resync to re-anchor a running client at a lower checkpoint #856 make a first-class flow), or lower the configured start height.
4. On restart, start_download sees scan_start = LOW, stored_filters_tip = HIGH-tip → preloads LOW.. → debug SIGABRT in SegmentCache::get_items; release silently returns sentinel filter data for the gap and match results over that range are wrong.

So "importing a wallet at a lower height" — the scenario suspected early in this thread — is a real trigger of the same crash, independent of anything our app did. That's why I'd argue against merging-then-reverting: the guard protects a legitimate user flow that's about to ship, and the release-mode half (silent wrong matches) exists for every variant of the state.

The path our device actually took (T1 + T2, both filed)

1. Wallet birth 0, filter sync stores 0..~1.5M, all healthy.
2. T1 (#894): mid-life client re-init from checkpoint 200000 → filter storage reset (tip=0) → refill resumed from the wallet's synced position (1,514,654) instead of the floor → island 1514654..tip. Session ends cleanly at committed 2,504,884 — still consistent, no crash.
3. T2 (dashpay/platform#4130 — agreed, it's a platform-wallet/swift-sdk issue): across an abrupt termination the wallet's committed height regresses (persisted syncedHeight=2504884 exists but isn't reapplied/stamped on the abrupt path) → next launch computes scan_start = max(birth 0, checkpoint 200000) = 200000 → below the island → crash-loop (each crash is itself an abrupt kill, re-arming T2).

On "hiding the underlying issue"

The discard path logs WARN Stored filters {start}..={tip} do not reach scan start {scan_start}; discarding them and re-downloading with the exact ranges — every future occurrence stays loudly visible in logs while both root causes (#894 upstream, platform#4130) are tracked and fixed. Without #893, the same state in a release build doesn't crash OR warn — it silently feeds wrong filter data to the matcher, which is the truly hidden failure mode.

🤖 Generated with Claude Code