test(drive-abci): rehearse a reproducible scheduled host fault and the existing recovery path

[DCG-Claude]

Issue being fixed or feature implemented

Part of the smart-contract plan in #4626 (workstream issue #4688, task FIX-03). Once contracts run inside blocks, three very different failures share one word: a guest that traps (a paid result), a node that cannot load a compiled module (that node stops), and a defect in how every node drives the scheduled phase (every block fails). The owner's selected policy for anything that changes execution or fees is the normal protocol upgrade, with no emergency pause, no activation-height override, no recovery binary and no guest-skip rule. Before the jobs phase is built, this PR keeps the three classes apart with tests that drive the real block-execution entry points, reproduces the third class at the scheduled-event integration point, rehearses the recovery path that exists for it, and reports the limitation of that path in a reference chapter.

Refs #4688

What was done?

Test-only fault hook at the scheduled-event integration point (packages/rs-drive-abci/src/config.rs, packages/rs-drive-abci/src/execution/engine/run_block_proposal/v0/mod.rs). PlatformTestConfig gains scheduled_event_host_fault: bool (default false in both Default and default_minimal_verifications()) and the constant SCHEDULED_EVENT_HOST_FAULT_MESSAGE. In run_block_proposal_v0, directly after run_dao_platform_events and before process_raw_state_transitions, a block gated by #[cfg(feature = "testing-config")] returns Err(Error::Execution(ExecutionError::CorruptedCodeExecution(SCHEDULED_EVENT_HOST_FAULT_MESSAGE))) when the flag is set. This follows the four existing testing_configs reads inside shipped v0 modules and the test_fault_injection seam in process_raw_state_transitions. Production builds (console,grovedbg,replay) never enable testing-config, so the compiled run_block_proposal_v0 is byte-identical; no vN generation, no version table, no fee schedule, no consensus error, no proto or SDK surface changes.

Unit tests through the dispatcher (packages/rs-drive-abci/src/execution/engine/run_block_proposal/mod.rs): should_fail_the_whole_proposal_at_the_scheduled_event_point_when_the_host_fault_is_armed pins that an empty proposal returns Err with the injected message (a whole-proposal failure, not a ValidationResult error and not a per-transition result); should_not_fail_the_proposal_at_the_scheduled_event_point_when_unarmed pins that the same proposal is valid with the flag off. The synthetic state carries an epoch 0 with its proposers tree and one validator set so the closing per-block events complete.

Strategy module (packages/rs-drive-abci/tests/strategy_tests/test_cases/scheduled_host_fault_tests.rs, registered in test_cases/mod.rs). All requests go through tenderdash_abci::Application on FullAbciApplication; twin platforms are built from one seed and compared block for block.

• paid_failure_stays_in_the_block_pays_and_the_chain_advances (class 1): the dashpay all-mutable contract with a skipped-position update at block 3 produces a PaidConsensusError with code 10411 and positive gas_used, and the chain reaches block 10 with verify_state_transition_results.
• node_local_fault_stops_only_that_node_and_it_commits_the_same_block_after_repair (class 2): twins A and B commit five blocks; B executes the next block; A, armed, fails it with the injected exception and its committed root is unchanged; disarmed, A re-executes the identical request to B's app hash and tx_results; both finalize through finalize_block and hold the same committed root and height.
• reproducible_scheduled_fault_halts_every_node_before_ordinary_transactions (class 3): both twins armed; every prepare_proposal and process_proposal fails on both nodes for two proposers, rounds 0 to 2, with and without transactions, while signalling latest+1, and for an epoch-change block; committed roots, heights, the versions counter (read with fetch_versions_with_counter) and next_epoch_protocol_version() are unchanged.
• execution_identical_hotfix_replays_the_faulted_proposal_and_the_chain_continues (class 3 recovery): control node C prepares and processes a concrete proposal R (non-empty, signalling latest+1, first block of epoch 1); A, armed, fails R as proposer and twice as validator with nothing left behind; A is disarmed and reopened from disk with TempPlatform::open_with_tempdir; A processes the unchanged R to C's app hash and tx_results; both finalize R, land in epoch 1 with the versions counter holding exactly {latest+1: 1} and the created identities present on both; continue_chain_for_strategy runs five more blocks on each and the committed roots stay equal.

CI gate (.github/workflows/tests-rs-workspace.yml): the two nextest filters now include test(~scheduled_host_fault) in the PR phase and exclude it from the push-only phase, next to the collision module; the comment block above them is extended. actionlint reports the same pre-existing shellcheck notes as on the base branch and nothing new.

Book chapter (book/src/architecture/block-failure-classes.md, listed in book/src/SUMMARY.md under Architecture): the three classes with their surfaces in rs-drive-abci and Tenderdash 1.7 (internal/consensus/block_executor.go, internal/consensus/prevoter.go, internal/consensus/replay.go, internal/state/execution.go, verified on tag v1.7.0), why a signalling upgrade cannot progress on a halted chain (vote written inside the block, tally on an epoch-change block, activation on a later block), the hotfix-and-replay recovery the tests rehearse, the historical evo1 gates and emergency parameter updates described as evidence of coordinated releases rather than a procedure, the limitation, how to run the rehearsal, and rules for the jobs phase.

How Has This Been Tested?

Locally on macOS (Darwin 24.6.0, arm64), each command redirected to a file with its exit code captured:

cargo fmt --all
cargo clippy -p drive-abci --all-features --all-targets -- --no-deps -D warnings
cargo check --workspace --all-targets
cargo test -p drive-abci --lib run_block_proposal
cargo test -p drive-abci --test strategy_tests scheduled_host_fault
cargo test -p drive-abci --test strategy_tests process_proposal_collision
cargo test -p drive-abci --test strategy_tests run_chain_insert_one_new_identity_and_a_contract_with_bad_update
cargo test -p drive-abci --test strategy_tests run_chain_with_temporarily_disabled_contested_documents
/tmp/actionlint .github/workflows/tests-rs-workspace.yml

Results: 5 run_block_proposal unit tests pass; the 4 scheduled host fault simulations pass in under two seconds; the collision, bad-update and voting simulations pass. The book builds in CI (book-preview.yml); mdbook is not installed locally.

Breaking Changes

None. No consensus, protocol, fee, storage or API change. The only library-visible additions are on the testing-config feature (PlatformTestConfig::scheduled_event_host_fault, SCHEDULED_EVENT_HOST_FAULT_MESSAGE); the one spread-less PlatformTestConfig literal outside config.rs (voting_tests.rs) gained the field.

Decisions taken (provisional values)

1. The scheduled-event integration point is the position after run_dao_platform_events and before process_raw_state_transitions in run_block_proposal_v0, the processing order the scheduling draft proposes (engineering convention, not an owner decision). When the jobs phase lands, the hook moves inside it.
2. The paid-guest-failure class is represented by its native analogue, a paid consensus error on a contract update, because no guest runtime exists on this base; the same test extends to a scheduled trap once jobs exist.
3. The node-local class is represented by one node failing process_proposal with the same exception a storage fault produces; the chain-lock Reject path is documented, not driven (driving it needs a second mock Core RPC with the harness's masternode data).
4. The class-3 fault surfaces as Err(ExecutionError::CorruptedCodeExecution); a panic in the same phase is documented as equivalent in network effect.
5. Tenderdash behaviour is cited from tag v1.7.0 sources (dashmate pins dashpay/tenderdash:1.7). Write-ahead-log replay is modelled as re-delivering the identical RequestProcessProposal to the restarted node; the proposer-side re-prepare is not part of replay.
6. The new strategy module joins the PR nextest gate next to the collision module (it runs in under two seconds); maintainers may move it to the push-only phase.
7. The chapter reports the execution-changing case as having no demonstrated progressing path under the selected policy; it does not propose one.
8. Hand-built blocks are proposed by the quorum member the simulation would pick next (walking the current quorum in key order), so the validator set does not rotate under the block and the continuation after recovery follows the harness's own proposer order.

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have added "!" to the title and described breaking changes in the corresponding section if my code contains any
• I have made corresponding changes to the documentation if needed

For repository code-owners and collaborators only

• I have assigned this pull request to a milestone

Dash-Tasks: FIX-03

🤖 Generated with Claude Code

---

🤖 Posted autonomously by DashVM (Claude Fable 5.1) on behalf of pasta.

Automated reviewer consensus (Fable 5.1 implementer, GPT-6 Astra reviewer)

Reviewer consensus

Plan Review consensus

• FIX-03-R1 [major] Do not treat historical exceptions as an authorized recovery policy -> resolved
  • at /Users/dashvm/work/wt/FIX-03/PLAN.md:245 — §3.3, item 6
• FIX-03-R2 [major] Replay the failed proposal after reopening the database -> resolved
  • at /Users/dashvm/work/wt/FIX-03/PLAN.md:206 — §3.2, hotfix recovery test
  • round 1 FIX-03-R1: accept: Section 3.3 item 6 and section 2 now describe the evo1 height gates as replay preservation of an already committed history and the emergency consensus params and lowered threshold as rules evaluated inside block processing that still needed blocks; none is presented as a procedure. The chapter repor
  • round 1 FIX-03-R2: accept: Section 3.2 replaces the continuation-only recovery test with a replay rehearsal: a concrete non-empty epoch-boundary proposal is prepared on the control, the process request is stored unchanged, the faulted node fails it twice, is disarmed and reopened from disk, replays the identical request, and

Review consensus

• FIX03-01 [nit] Use an allowed scope for the documentation commit -> noted
  • at .github/workflows/pr.yml:34; commit 3c7a94e

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: aa02d51a-46dd-4966-b084-f88bb439b218

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📖 Book Preview built successfully.

Download the preview from the workflow artifacts.
To view locally: download the artifact, unzip, and open index.html.

Updated at 2026-09-17T22:49:16.258Z

[thepastaclaw]

⚠️ DEGRADED — Final review complete — no blockers (commit 3c7a94e) · triage: normal · stand-in models (primary models out of quota)

[codecov]

Codecov Report

❌ Patch coverage is 46.45161% with 83 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.40%. Comparing base (5f1e0cc) to head (3c7a94e).

Files with missing lines	Patch %	Lines
...bci/src/execution/engine/run_block_proposal/mod.rs	50.38%	65 Missing ⚠️
packages/rs-drive-abci/src/config.rs	9.09%	10 Missing ⚠️
.../src/execution/engine/run_block_proposal/v0/mod.rs	38.46%	8 Missing ⚠️

❌ Your patch check has failed because the patch coverage (46.45%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@              Coverage Diff              @@
##           v5.0-dev    #4808       +/-   ##
=============================================
- Coverage     86.36%   75.40%   -10.97%     
=============================================
  Files          2766     2799       +33     
  Lines        366105   411517    +45412     
=============================================
- Hits         316191   310296     -5895     
- Misses        49914   101221    +51307     

Components	Coverage Δ
dpp	75.13% <ø> (-12.16%)	⬇️
drive	76.86% <ø> (-7.39%)	⬇️
drive-abci	73.60% <46.45%> (-16.06%)	⬇️
sdk	∅ <ø> (∅)
dapi-client	∅ <ø> (∅)
platform-version	∅ <ø> (∅)
platform-value	86.67% <ø> (-6.25%)	⬇️
platform-wallet	∅ <ø> (∅)
drive-proof-verifier	34.71% <ø> (-15.07%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[DCG-Claude]

The codecov patch and project numbers on this head are a reporting artifact, not missing coverage, so I am not rerunning the job or pushing for it.

What the numbers rest on: the Rust workspace job for 3c7a94e ran on the self-hosted mac-runner-pasta and passed all 15515 tests, including the two new run_block_proposal unit tests (which execute the hook lines codecov lists as missed in run_block_proposal/v0/mod.rs) and the four scheduled_host_fault simulations. The file reports codecov built from that run map profiles onto line tables that do not belong to this branch: run_block_proposal/v0/mod.rs is instrumented up to line 551 while the file is 485 lines here, and config.rs up to line 1273 while the file is 1249 lines. Inside the patch that shows up as a struct literal with one field "hit" and its neighbours "missed", which cannot happen in a single test run. The same shifted mapping is what drives the project-wide drop on files this PR does not touch (for example rs-dpp/src/block/finalized_epoch_info/mod.rs reports 78 misses inside functions the job log shows passing).

This is the known self-hosted-runner line-table mismatch seen on other v5.0-dev PRs; a rerun adds a second upload with the same stale mapping. Neither codecov check is in the v5.0-dev rulesets.

---

🤖 Posted autonomously by DashVM (Claude Fable 5.1) on behalf of pasta.

[thepastaclaw]

⚠️ DEGRADED — Final validation — Phase 1 + Phase 2

⚠️ DEGRADED review. The primary review models were unavailable (gpt-6-astra unavailable: All credentials for model gpt-6-astra are cooling down (last error: usage_limit_reached: The usage limit has been reache), so this review ran on stand-in models: gpt-5.6-luna → muse-spark-1.3-contributor, gpt-5.6-sol → muse-spark-1.3-contributor, gpt-5.6-terra → muse-spark-1.3-contributor, gpt-6-astra → muse-spark-1.3-contributor. Both review phases and the independent verifiers still ran, but on weaker models, with Phase 1 capped at high effort. Treat the verdict as provisional; a full-strength re-review will run on the next push once the primary models are back.

Test-only rehearsal of the three block-failure classes with no production impact. I independently verified the safety claims: the fault hook in run_block_proposal_v0 and the new PlatformTestConfig field plus message constant are all #[cfg(feature = "testing-config")] gated, default to false in both constructors, and testing_configs is #[serde(skip)] so the flag is unreachable from config files. No version-table, consensus-error, fee, serialization, or wire-format changes. No in-scope findings from any lane.

Review provenance

Source: reviewer 1: muse-spark-1.3-contributor (agent: phase1-reviewer, role: general); reviewer 2: muse-spark-1.3-contributor (agent: phase1-reviewer, role: architecture-layering); reviewer 3: muse-spark-1.3-contributor (agent: phase1-reviewer, role: platform-versioning); reviewer 4: muse-spark-1.3-contributor (agent: phase1-reviewer, role: rust-quality); reviewer 5: muse-spark-1.3-contributor (agent: phase1-reviewer, role: security-auditor); reviewer 6: muse-spark-1.3-contributor (standing in for gpt-6-astra) (agent: phase2-reviewer, role: general); reviewer 7: muse-spark-1.3-contributor (standing in for gpt-6-astra) (agent: phase2-reviewer, role: architecture-layering); reviewer 8: muse-spark-1.3-contributor (standing in for gpt-6-astra) (agent: phase2-reviewer, role: platform-versioning); reviewer 9: muse-spark-1.3-contributor (standing in for gpt-6-astra) (agent: phase2-reviewer, role: rust-quality); reviewer 10: muse-spark-1.3-contributor (standing in for gpt-6-astra) (agent: phase2-reviewer, role: security-auditor); final verifier: muse-spark-1.3-contributor (standing in for gpt-6-astra) (agent: astra-verifier, role: final-verifier)

• Degraded mode: gpt-6-astra unavailable: All credentials for model gpt-6-astra are cooling down (last error: usage_limit_reached: The usage limit has been reache (detected by probe, since 2026-09-18T05:22:01Z); stand-ins gpt-5.6-luna → muse-spark-1.3-contributor, gpt-5.6-sol → muse-spark-1.3-contributor, gpt-5.6-terra → muse-spark-1.3-contributor, gpt-6-astra → muse-spark-1.3-contributor; Phase 1 effort capped at high
• Triage: normal by muse-spark-1.3-contributor (standing in for gpt-6-astra) (effort low) — Large test-only fault hook gated behind testing-config plus docs and strategy tests with no production consensus, funds, crypto, or migration change.
• Phase 1 reviewers: muse-spark-1.3-contributor — general (completed, effort high); agent phase1-reviewer, muse-spark-1.3-contributor — architecture-layering (completed, effort high); agent phase1-reviewer, muse-spark-1.3-contributor — platform-versioning (completed, effort high); agent phase1-reviewer, muse-spark-1.3-contributor — rust-quality (completed, effort high); agent phase1-reviewer, muse-spark-1.3-contributor — security-auditor (completed, effort high); agent phase1-reviewer
• Phase 1 model: muse-spark-1.3-contributor — not quota-gated; passed over gemini-3.8-flash-high (lane failed), glm-5.3-flash (zai below 15% reserve: 5h 99% left, weekly 14% left)
• Fresh verifier: muse-spark-1.3-contributor (standing in for gpt-6-astra) — final-verifier; agent astra-verifier
• Phase 2 reviewers: muse-spark-1.3-contributor (standing in for gpt-6-astra) — general (completed, effort high); agent phase2-reviewer, muse-spark-1.3-contributor (standing in for gpt-6-astra) — architecture-layering (completed, effort high); agent phase2-reviewer, muse-spark-1.3-contributor (standing in for gpt-6-astra) — platform-versioning (completed, effort high); agent phase2-reviewer, muse-spark-1.3-contributor (standing in for gpt-6-astra) — rust-quality (completed, effort high); agent phase2-reviewer, muse-spark-1.3-contributor (standing in for gpt-6-astra) — security-auditor (completed, effort high); agent phase2-reviewer

Out-of-scope follow-up suggestions (1)

These are valid observations, but they are outside this PR's scope and should be handled in separate issues or author/maintainer-requested PRs rather than blocking this review.

• Real scheduled-jobs phase will need full versioning — This PR deliberately rehearses with a test hook before any guest runtime exists; when the jobs phase lands (hook moves inside it, paid trap results, fee schedule, possible new transitions/errors), that work will need a new vN implementation, dispatcher arm, PlatformVersion slots, activation gates, and estimation/execution/proof coverage.
  • Follow-up: Track the versioned rollout of the jobs phase in the smart-contract workstream (issue #4688) when execution or fees change.