Use REST spenttxouts for block indexing, plus consistency fixes and RocksDB multi-CF#217
Open
shesek wants to merge 13 commits into
Open
Use REST spenttxouts for block indexing, plus consistency fixes and RocksDB multi-CF#217shesek wants to merge 13 commits into
shesek wants to merge 13 commits into
Conversation
Remove the V1-to-V2 migration script. This branch makes several DB schema changes and no longer supports a migration path for old DBs, requiring a reindex instead.
Remove blk*.dat support and the `FetchFrom` dispatch around it, leaving support for RPC-based syncing only. Drop the now-obsolete `--blocks-dir` and `--jsonrpc-import` config, daemon BLK-file helpers and initial-sync switching logic.
Drop light-mode indexing and its RPC-based querying. The indexer now always stores the data needed for local block and transaction queries. This changes the compatibility marker format which breaks DB compatibility, but doesn't bump `DB_VERSION`. It is bumped once in a later commit, when switching to column families.
Consume prevouts as ordered iterables during block indexing instead of building a `HashMap` and looking up each prevout by outpoint. This avoids extra map build/lookup work and matches the shape needed later to consume the `spenttxouts` format. The shared shape is one ordered prevout vector per block: txstore-based lookup partitions the flat batch `multi_get` result into that shape, and the `spenttxouts` mode will flatten Bitcoin Core's per-block/per-transaction results into the same shape.
Split the batch-level add/index helpers into `add_block()` and `index_block()` to make the row collectors reusable by the later `spenttxouts` indexing mode.
Have `headers_to_process()` return `HeaderWork` entries recording whether each header still needs txstore rows, history rows, or both. This avoids taking another lock to re-check block status after fetching blocks, and will be shared by both indexing modes.
Remove pre-existing unnecessary cloning. This cleanup is unrelated to the `spenttxouts` changes and only affects the existing indexing mode.
shesek
changed the title
- Switch block fetching to the binary REST `block` endpoint, making REST the only supported block fetch source for both Bitcoin Core and Elements. - Add REST client support for `spenttxouts`, currently unused until the `spenttxouts` indexing mode is added in the next commit. - Reuse the bounded daemon Rayon pool for parallel REST requests. - Start test regtest nodes with REST enabled.
The new mode fetches each block's spent prevouts using binary REST, avoids local TXO lookups during indexing, and allows blocks to be processed in parallel without chain-order dependencies. Bitcoin Core indexing now defaults to the spenttxouts-based mode, with `--no-spenttxouts` as a fallback for older releases that don't support it (<v30). Elements stays on the existing txstore-based indexing mode. The integration tests node was updated to Bitcoin Core v30.2.
…o prevent OOM With pin_l0_filter_and_index_blocks_in_cache enabled, each L0 SST file pins its bloom filter and index blocks in the block cache. With the old settings (trigger=64, 256 MB write buffers), L0 could accumulate to the stop threshold — at ~9.75 MB pinned per file, this consumed ~3.7 GB across 3 DBs, overflowing the 2 GB block cache and spilling to heap, triggering OOM during mainnet initial sync around block 750k. Lower the trigger to 32 and set slowdown=×3 (96 files) and stop=×4 (128 files). With 128 MB write buffers (~4.9 MB filter blocks per file), pinned metadata at the stop threshold is ~1.88 GB across 3 DBs — within the 2 GB cache at all times. Set the write buffer CLI flag accordingly: --db-write-buffer-size-mb=128
Since the spenttxouts-based indexer can process blocks in parallel and complete them out of order, consistency now requires crash recovery based on completion markers and revised tip-publication rules: - Derive startup chain visibility from the persisted `t` chain plus txstore/history completion markers. The first daemon-aware `update()` then sweeps history-complete stale blocks before normal indexing work. - Change reorg handling to persist the common ancestor in the DB prior to undoing stale block data, so a crash during stale cleanup restarts from the rollback point and completes recovery. - Store the block height alongside the block header in `B` rows, so that stale recovery can rebuild `HeaderEntry`s from RocksDB alone. This changes the DB schema but does not bump `DB_VERSION`. It is bumped once in a later commit, when switching to column families. - Add an indexer consistency and crash recovery document describing the current safety model, recovery assumptions and caveats. At this point stale cleanup still depends on the daemon for stale block contents, and using separate RocksDB databases can still lead to a corrupted state because cross-DB WAL-disabled writes may be flushed implicitly and independently.
Reconstruct stale `BlockEntry`s and spent prevouts from local txstore `B`, `X`, `T`, and `O` rows instead of fetching them from the daemon. This is used for both live reorg cleanup and startup stale cleanup, so undo no longer depends on the currently connected daemon having the stale block.
- Store `default`, `txstore`, `history`, and `cache` as column families in one physical RocksDB. - Move compatibility metadata and persisted tip `t` into the default column family. - Flush txstore/history bulk writes with RocksDB atomic flush before publishing `t`, replacing the separate-DB flush sequence and removing the remaining cross-DB durability caveat. - Bump `DB_VERSION` for the schema changes in this branch.
shesek
force-pushed
the
202604-indexer-spenttxouts
branch
from
02b2052 to
39d3ca3
Compare
Collaborator
Author
|
Shoutout and a big thanks to @romanz that made this possible! 🙌 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a new indexing mode using the Bitcoin Core
spenttxoutsREST endpoint to look up spent prevouts and switches to REST for raw block contents (in both indexing modes).The existing indexing mode operates in two phases: first populate
txstore, then populatehistorywhile looking up spent prevouts intxstore. This makes indexing read-heavy and introduces an ordering requirement: before a block can be indexed intohistory, it and all its ancestors must first be populated intotxstore.The new indexer fetches each block alongside its spent prevouts directly from the Bitcoin Core binary REST API using the
blockandspenttxoutsendpoints, then writes the completetxstore+historyDB rows for each block. This allows blocks to be indexed in any order with full parallelism, as a write-only workload with no local RocksDB reads.The new indexer is now the default operating mode for Bitcoin, but the old indexer is still used when
spenttxoutsis not supported: with Elements/Liquid where it is the only mode, or with Bitcoin Core <v30 using--no-spenttxouts.Commits can be grouped into:
Cleanup: removes obsolete operating modes that are no longer supported, to reduce maintenance burden when making indexer/schema changes. This includes DB migration, BLK import support and light-mode. (fca55c0, 78c78d8, 665bcb3)
Groundwork: prepares shared infrastructure for both indexing modes: switch to ordered prevout consumption, extract reusable code and introduce
HeaderWork, plus a perf fix. (6a96e89, e05a8a1, f5b89d2, e51cef2)Core work: adds binary REST client support and the
spenttxouts-based indexer, and tunes RocksDB for increased write throughput. (5919b3f, fbb6b23, c779804)Consistency: adapts crash recovery, tip publication, stale undo and RocksDB durability for out-of-order block indexing. (a998233, 25d5387, 39d3ca3)
Consistency and crash recovery
Out-of-order block indexing required adapting crash recovery to use a new startup stale cleanup mechanism, where Electrs compares the daemon's best chain with the
Dcompletion markers and cleans up any stale blocks before normal indexing work. During reorgs, the common ancestortis now persisted before stalehistorydeletion so a mid-way crash will recover from a safe tip and complete cleanup.Store::open()also includes a startup fail-safe mechanism from an earlier recovery design (that was discarded), that trims chain visibility down to the fully indexed contiguous prefix. The current design does not depend on this, but I kept it anyway to guard against unexpected/invalid state. This could alternatively panic instead of trim.Stale undo now uses local RocksDB data instead of the current bitcoind. This makes recovery safe in multi-backend deployments, where Electrs might reconnect to a different daemon that does not have the stale block.
The PR also adds an AI-assisted document describing the safety model, enumerating potential crash-recovery failure cases, and explaining how they are handled. This helped validate the implementation and identified the two (pre-existing) failure cases fixed by 25d5387 and 39d3ca3. I wasn't sure whether to include this in the final PR, but ended up keeping it as reference for humans/AIs making consistency/safety-sensitive changes.
RocksDB Column Families
The index now uses a single RocksDB database with four column families:
default,txstore,historyandcache.This was necessary to resolve a consistency/durability issue with the prior multi-DB design, and also desirable regardless.
Potential Followups
spenttxoutssupport (a simple+109patch to rest.cpp plus docs/tests), we could consider supporting recent Core/Elements versions only and remove the old indexer mode entirely.bitcoin_slicescrate for allocation-free block parsing. I initially thought to include this alongside the new indexer, but it's big enough of a change to warrant its own PR. We could take inspiration frombindexwhich usesbitcoin_slices.Backward Compatibility
This PR breaks DB compatibility and requires a complete reindex.