Skip to content

perf(producer): admit inserts in batches and read their nonces outside the mempool lock - #4638

Open
yzang2019 wants to merge 16 commits into
yzang/evmonly-settled-noncefrom
yzang/producer-admission-batching
Open

yzang2019 wants to merge 16 commits into
yzang/evmonly-settled-noncefrom
yzang/producer-admission-batching

Conversation

@yzang2019

@yzang2019 yzang2019 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #4633 and carries #4608 as a merge; review the commits after the merge.

Problem

  • On a single Autobahn validator the producer, not execution, capped throughput.
  • Every waiting InsertTx contended for the one mempool lock and read its sender's nonce from the app while holding it; prune did the same for every unsettled sender of each pruned block, starving the producer loop that frees capacity.
  • SettledNonce also scanned the in-flight commit's whole change list on every read.

Change

  • poolCanAdmit refuses a full or closed session from atomics, without the lock. Inserts waiting for a batch count toward MaxPendingInserts.
  • Waiting inserts join a per-session queue. The call holding the admission turn runs one batch (admitNextBatch) and hands the turn to the oldest insert queued meanwhile, so no call drains arrivals indefinitely; a call that gives up while queued leaves the queue.
  • Each batch reads its untracked senders' nonces in one EvmNonces call outside the lock (Executor.SettledNonces: one view, parallel reads, served first from a per-commit address index), then appends in FIFO order under one lock. A prune since the read makes the batch read again outside the lock.
  • pruneMempool reads a pruned block's unsettled senders in one batch. The non-waiting path is a single try that never joins the queue.

Impact

  • Single validator: throughput rose about 3.5× and execution became the limit.

Risk

  • Admission order and the nonce checks are unchanged; only where and how often nonces are read changes.
  • When the mempool is full, waiters admitted by a prune still read stale senders' nonces under the lock; that path stayed cold with lane capacity 300+ and is tracked as a follow-up (STO-852).

Tests

  • New: a cancelled queued insert never holds the turn, concurrent inserts all return, a prune during a batch's nonce read is re-read outside the lock, and a closed session fails its queued inserts without a nonce read.
  • producer, evmonlyapp, proxy and giga/evmonly pass with -race.

yzang2019 and others added 11 commits October 23, 2026 01:21
Blocked InsertTx calls were admitted one at a time, each woken by the one before it, and new calls queued behind them even with room in the mempool. Once a queue formed, admission ran at goroutine hand-off speed and the queue never drained under load. Whoever frees capacity now appends the queued txs in arrival order until the mempool is full again.

Co-authored-by: Cursor <cursoragent@cursor.com>
…t of the queue, drop the wakeups metric

Co-authored-by: Cursor <cursoragent@cursor.com>
…rain' into yzang/producer-admission-batching

* origin/yzang/autobahn-admission-batch-drain:
  autobahn: reset pending inserts on close, keep non-waiting inserts out of the queue, drop the wakeups metric
  autobahn: admit the whole insert queue when lane capacity frees
…full without taking the mempool lock

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…ew in parallel

Co-authored-by: Cursor <cursoragent@cursor.com>
…batch

Co-authored-by: Cursor <cursoragent@cursor.com>
…une makes them stale

Co-authored-by: Cursor <cursoragent@cursor.com>
… batched admission per insert

Co-authored-by: Cursor <cursoragent@cursor.com>
@cursor

cursor Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches mempool admission ordering, concurrency, and EVM nonce validation paths; semantics are intended to stay the same but the new batching and lock-free pre-checks add coordination edge cases covered by new tests.

Overview
Autobahn producer no longer admits each waiting InsertTx one-by-one under the mempool lock with per-tx app nonce reads. Blocking inserts join a session admission queue; one caller at a time runs admitBatch, which loads untracked EVM senders via EvmNonces outside the lock, then appends in FIFO order under a single lock. poolCanAdmit for the blocking path uses atomics (closed, pending + queued counts) so overload does not starve the prune path that frees capacity. Prune settles unsettled senders with one EvmNonces call; capacity frees drainWaiters to admit multiple queued txs at once. The capacity_wait_wakeups metric is removed.

EVM-only executor adds SettledNonces (one store view, parallel reads, pipelineNonces index for in-flight commits) and SettledNonce uses that index instead of scanning the pipeline change list. Proxy / evmonlyapp expose EvmNonces with a fallback to repeated EvmNonce.

Reviewed by Cursor Bugbot for commit 602e9d4. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedOct 27, 2026, 1:34 AM

@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.82927% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.48%. Comparing base (3ef3577) to head (602e9d4).

Files with missing lines Patch % Lines
...i-tendermint/internal/autobahn/producer/mempool.go 92.30% 12 Missing ⚠️
sei-tendermint/internal/proxy/proxy.go 0.00% 7 Missing ⚠️
sei-tendermint/internal/evmonlyapp/app.go 0.00% 6 Missing ⚠️
giga/evmonly/giga_store.go 93.93% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@                       Coverage Diff                       @@
##           yzang/evmonly-settled-nonce    #4638      +/-   ##
===============================================================
+ Coverage                        59.45%   59.48%   +0.02%     
===============================================================
  Files                             2038     2038              
  Lines                           160009   160140     +131     
===============================================================
+ Hits                             95139    95256     +117     
- Misses                           64865    64879      +14     
  Partials                             5        5              
Flag Coverage Δ
sei-chain 57.46% <86.82%> (+0.02%) ⬆️
sei-db 75.17% <ø> (ø)
sei-db-state-db 80.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
giga/evmonly/executor.go 89.93% <ø> (ø)
.../internal/autobahn/producer/metrics/metrics.gen.go 0.00% <ø> (ø)
...mint/internal/autobahn/producer/metrics/metrics.go 0.00% <ø> (ø)
sei-tendermint/internal/autobahn/producer/state.go 83.67% <100.00%> (-1.03%) ⬇️
giga/evmonly/giga_store.go 87.36% <93.93%> (+0.99%) ⬆️
sei-tendermint/internal/evmonlyapp/app.go 85.19% <0.00%> (-1.26%) ⬇️
sei-tendermint/internal/proxy/proxy.go 53.84% <0.00%> (-4.49%) ⬇️
...i-tendermint/internal/autobahn/producer/mempool.go 92.61% <92.30%> (-1.07%) ⬇️

... and 27 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

seidroid[bot]
seidroid Bot previously requested changes Oct 26, 2026

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR batches waiting InsertTx calls into a per-session admission queue. One caller drains the queue, reading each batch's untracked sender nonces outside the mempool lock (one EvmNonces call, served first from a per-commit nonce index) and then appending under a single lock; prune also reads its unsettled senders in one batch. The blocker: the caller that starts admission keeps draining the queue for as long as new inserts arrive, so under the sustained load this PR targets its own InsertTx can block indefinitely and ignore context cancellation. I reviewed against the PR's merge tree (f87cc22). Both codex findings hold and are kept: the leader one is the blocker and the queue-bound one is a suggestion. My own addition is that queued txs drained after a prune still read nonces one at a time under the lock.

Blocking

1 finding on the changed lines, as inline comments.

Non-blocking

2 findings on the changed lines, as inline comments.

  • No test covers a lead caller whose context is cancelled while it is draining, or admitBatch's retry when a prune moves first between the nonce read and the locked append. Both are new paths and easy to break silently.

seidroid review · decision request_changes · session 9c9df99d1207408a92280fbf85efdc3a · turn resp_claude_2ea0520bc88e0d876a4015f38b9bd0ac · item d9b7f16dd7f45d6d8928b3604409c2a2

Findings: 1 blocking | 3 non-blocking | 3 posted inline

Comment thread sei-tendermint/internal/autobahn/producer/mempool.go Outdated
Comment thread sei-tendermint/internal/autobahn/producer/mempool.go Outdated
Comment thread sei-tendermint/internal/autobahn/producer/mempool.go
…ion-batch-drain

* origin/main: (25 commits)
  evmonly: reuse one EVM and state DB per OCC worker, claim ranges, merge in parallel (#4610)
  giga: add giga.execution.gc_percent to raise the Go GC target (#4609)
  feat(giga-state-sync): add the state sync switch and the Autobahn layout (#4560)
  autobahn: export lane block and proposal latency by consensus step (#4606)
  Remove the wasm CLI commands and the generated [wasm] config (#4634)
  StateWAL emits Mutation objects DO NOT MERGE INTO 6.8 (#4628)
  Remove the unused x/store and x/evm/artifacts root packages (#4635)
  Remove unused contracts from example/ (#4630)
  Remove unused scripts and sei-wasmd contrib leftovers (#4624)
  writing qc + blocks atomically (#4599)
  Fix broken links and stale .PHONY names left by the cleanups (#4626)
  Remove support for pre-cancun blocks, simplify storage logic (#4575)
  Remove the unused upstream Tendermint e2e harness (#4623)
  Record sei-cosmos mockgen commands as go:generate directives (#4622)
  Move cosmovisor out of sei-chain into sei-protocol/cosmovisor (#4621)
  Remove the Cosmos Swagger UI and sei-cosmos contrib tooling (#4619)
  Remove the unused sei-cosmos, sei-tendermint and sei-wasmd Makefiles (#4620)
  Validate snapshot __metadata against the commit being published (#4613)
  flatKV audit command (#4600)
  Remove the vendored Tendermint spec (#4617)
  ...
yzang2019 and others added 3 commits October 26, 2026 18:16
…o yzang/producer-admission-batching

* origin/yzang/evmonly-settled-nonce:
  perf(evmonly): open the view only when the in-flight commit lacks the sender's nonce
  evmonly: reuse one EVM and state DB per OCC worker, claim ranges, merge in parallel (#4610)
  giga: add giga.execution.gc_percent to raise the Go GC target (#4609)
  feat(giga-state-sync): add the state sync switch and the Autobahn layout (#4560)
  autobahn: export lane block and proposal latency by consensus step (#4606)
  Remove the wasm CLI commands and the generated [wasm] config (#4634)
  StateWAL emits Mutation objects DO NOT MERGE INTO 6.8 (#4628)
  Remove the unused x/store and x/evm/artifacts root packages (#4635)
  Remove unused contracts from example/ (#4630)

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	giga/evmonly/giga_store.go
…rain' into yzang/producer-admission-batching

* origin/yzang/autobahn-admission-batch-drain:
…, and count queued admissions in the pending limit

Co-authored-by: Cursor <cursoragent@cursor.com>
@yzang2019

Copy link
Copy Markdown
Contributor Author

On the tests the seidroid review asked for, in 62166aa: TestInsertTx_CancelledQueuedInsertNeverHoldsTheTurn covers a caller that gives up while queued, TestInsertTx_ConcurrentInsertsAllReturn covers sustained concurrent arrivals, and TestInsertTx_PruneDuringNonceReadRereadsOutsideTheLock prunes a block during the batch's nonce read and checks the read is redone once outside the lock.

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This update (merge commit 047335f) makes the caller holding the admission turn run one batch and then hand the turn to the oldest queued insert, counts admitQueue toward MaxPendingInserts, and adds tests for a cancelled turn, concurrent inserts and a prune during the nonce read. That fixes my leader blocker and the queue-bound finding, so nothing blocks; the locked nonce read when a prune drains waiters is acknowledged as a follow-up. Of codex's two new readings, I kept the close one as a suggestion. I dropped the bound one because the batch it leaves uncounted is limited to roughly one admitQueue's worth, and anything past the limit is still rejected with errPendingFull once the batch reaches the lock. I could not run the tests here because there is no Go toolchain.

Non-blocking

2 findings on the changed lines, as inline comments.

seidroid review · decision approve · session 9c9df99d1207408a92280fbf85efdc3a · turn resp_claude_9aa981ccd5994a194c8a7f2505fa501c · item 14eef540feca570f98e1774db3fb0b99

Findings: 0 blocking | 2 non-blocking | 2 posted inline

for admitted < len(m.waiters) && !m.closed && !m.IsFull() {
t := m.waiters[admitted]
admitted++
t.finish(s.appendTx(m, ctrl, t.tx, t.resp, t.gasWanted, t.gasEstimated, t.appNonce, t.first))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — This still holds and you have agreed to a follow-up. Waiters drained by a prune always carry a stale t.first, so appendTx reads each untracked sender's nonce under the mempool lock one at a time, which is the pattern this PR removes everywhere else. Worth tracking so it is not lost once lane capacity or load changes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tracked as Linear STO-852 (batch the nonce reads of waiters a prune admits), so it isn't lost when lane capacity or load changes.

// close ends the session and releases every blocked InsertTx call.
func (m *mempoolInner) close(ctrl *utils.WatchCtrl) {
// close ends the session and fails every blocked InsertTx call with ErrNotProducing.
func (mp *mempool) close(m *mempoolInner, ctrl *utils.WatchCtrl) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion — (Raised by codex.) close fails only m.waiters. Tickets still in admitQueue learn of the close only when a batch reaches the mempool lock, so they wait out any in-flight EvmNonces read. The next leader also issues a nonce read for a session that is already closed. A cheap fix is for admitNextBatch or readUntrackedNonces to check mp.closed first and finish the batch with ErrNotProducing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 602e9d4: admitBatch checks mp.closed first and finishes every non-cancelled ticket in the batch with ErrNotProducing, so a closed session's queued inserts no longer wait out a nonce read and the next turn holder issues none. TestInsertTx_CloseFailsTheInsertsWaitingForABatch closes the session with two inserts queued and checks both fail without an EvmNonce call.

@seidroid
seidroid Bot dismissed their stale review October 27, 2026 01:24

Superseded: the latest review found nothing blocking in this change.

…e read

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant