Webhook sink: auth, signature, resumable delivery, reorg notifications, batching - #924
Draft
sduchesneau wants to merge 5 commits into
Draft
Webhook sink: auth, signature, resumable delivery, reorg notifications, batching#924sduchesneau wants to merge 5 commits into
sduchesneau wants to merge 5 commits into
Conversation
A receiver had no way to tell the sink's POST from anyone else's. The header value and the signing secret come from environment variables so they never appear on the command line.
🔍 Vulnerabilities of
|
| digest | sha256:579a47aa4abf8fef160c7028685e658b863818ae3c679aeda19aa2ab861e4282 |
| vulnerabilities |     |
| platform | linux/amd64 |
| size | 124 MB |
| packages | 380 |
📦 Base Image ubuntu:24.04
| also known as |
|
| digest | sha256:1e0a86e57d247923571b75e0aaf48a1449cf8c543d51fb3e07a4a7d7bfa79316 |
| vulnerabilities |   |
Dropping a block after the retries run out is silent data loss, and retrying in-process holds a Substreams stream open for nothing. With --webhook-on-failure=exit the block waits on disk and is delivered before the next stream is opened, so a dead endpoint costs no egress while it is down.
With a buffer, an undo below the first buffered block was dropped when nothing had been emitted yet. That is the shape of a restart from a cursor on a fork: the previous run emitted the forked blocks, and the handler never heard they were gone.
Until now a receiver was sent blocks that later left the chain and never heard about it. The notification is opt-in because it changes what a receiver must handle, and goes to its own URL so the block endpoint keeps a single payload shape.
sduchesneau
force-pushed
the
feature/webhook-sink-delivery
branch
from
September 4, 2026 13:55
dc3fe0b to
24f3d70
Compare
One call per block caps throughput at the receiver's round trip. With --webhook-batch-max-blocks the sink sends up to N blocks per call in a batch shape, opt-in so existing receivers keep the single-block shape.
sduchesneau
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five independent commits to
substreams sink webhook, in review order:--webhook-auth-header-name/--webhook-auth-header-value-envvarsend a verbatim header;--webhook-signing-secret-envvaraddsX-Substreams-Signature: t=<unix>,v1=<hex>over<t>.<body>. Secrets come from env vars so they never sit on the command line.Client.Callnow returns a*DeliveryErrorwith status and attempt count once retries run out.--webhook-on-failure=exitwrites the failed payload to<state-file>.pending, writes a JSON reason to--webhook-termination-logwhen that file exists, and exits with status 75. On the next start the pending payload is delivered before any Substreams stream is opened, so retrying against a dead endpoint costs no egress. The file is only written on failure: a kill mid-call leaves nothing behind, the cursor was not advanced, and the stream re-sends that block. A changed URL or secret resetsfirst_attempt_at. Also exposessubstreams_sink_progress_block.--undo-buffer-size, an undo signal reaching below the buffer while nothing had been emitted yet was swallowed. That is what a restart from a cursor on a fork produces, so the previous run's forked blocks were never rolled back. The buffer now reports whether it absorbed the undo and the sinker forwards it to the handler otherwise. Affects every buffered sink, not just the webhook one.--webhook-undo-urlreceives{"lastValidBlock": {"number", "id"}, "manifest": {"moduleName"}}per undo signal, with the same headers, retry, on-failure and pending-file rules as blocks. Opt-in; without it the cursor moves back and only replacement blocks are delivered, as before.--webhook-batch-max-blocks=Nswitches every call to{"manifest": {...}, "blocks": [{"clock", "data"}, ...]}with up to N blocks. A batch is sent when full, when--webhook-batch-max-wait(1s) has passed and the next block arrives, when the chain is live, before an undo notification, and when the stream ends. A failed batch is kept and resumed as one payload. Toggling batching while the sink is stopped discards a pending payload of the other shape and lets the stream re-send its blocks. Off by default.Not included: a receive-window cap on the Substreams client.
Design context: streamingfast/services-control-plane#61.