Run DKG on main chain#707
Draft
jannikluhn wants to merge 81 commits into
Draft
Conversation
Adds mise-test-setup/e2e-tests/ as a sibling mise project so the human-facing mise-test-setup remains untouched. Env vars and parent tasks (clean, wait-for-dkg, test-decryption, etc.) are inherited via mise's directory-walking config discovery — verified that `mise run clean` from e2e-tests/ resolves to the parent task without modification. The shared helper e2e_utils.py provides: - wait_for_dkg_success(keyper_set_index=N, timeout=120): Delegates to `mise run wait-for-dkg --keyper-set-index N` with a subprocess timeout. The existing task is eon-aware and polls through prior failures of the same set, which is what the happy-path test needs across keyper set transitions. - wait_for_dkg_success(timeout=120): (no keyper_set_index) Polls dkg_result for any success='t' row. Used by the offline-recovery test after triggering a retry, where the test cares only that DKG eventually succeeded. - wait_for_dkg_failure(timeout=90): Polls dkg_result for any success='f' row. Used by the offline-recovery test to assert DKG actually failed below threshold rather than just stalling. All timeouts raise SystemExit with a clear message so the calling task exits nonzero. Behavior verified with mocked DB queries covering all five paths (success-timeout, success-on-t, failure-timeout, failure-on-f, success-ignores-f). Implements issue 001-harness-scaffold.md. Unblocks 002 and 003. Files changed: - mise-test-setup/e2e-tests/mise.toml (new) - mise-test-setup/e2e-tests/mise-tasks/e2e_utils.py (new) Next iteration: implement test-dkg-happy-path (002) and test-dkg-offline-recovery (003), then the test runner (004). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements `test-dkg-happy-path` in `mise-test-setup/e2e-tests/mise-tasks/`. The task starts from a clean environment, runs the initial DKG for keyper set 1 (indices 0,1,2, threshold 2), then transitions through three more keyper sets: 0,1,2,3 / threshold 3, the zero-overlap set 3,4,5 / threshold 2, and finally back to 0,1,2 / threshold 2. After each successful DKG, decryption is verified via `test-decryption`. Key decisions: - `NUM_KEYPERS=6` is exported by the test script itself (not via `mise-test-setup/mise.toml`), keeping the parent defaults untouched per the PRD. Six keypers are needed so the 3,4,5 replacement set has enough distinct indices. - DKG waits for sets 2-4 go through `wait_for_dkg_success(keyper_set_index)` from the shared helper, which subprocess-wraps `mise run wait-for-dkg --keyper-set-index N` with a 120s timeout so a stalled DKG fails fast. - Set 1 uses `wait-for-initial-dkg` directly, since that task already chains the `up` + `add-initial-keyper-set` setup it needs. - Parent tasks are invoked by name (`mise run <task>`); mise's directory-walking discovery resolves them via the parent `mise.toml`, and task working-dir resolution keeps the docker compose context correct. Files changed: - mise-test-setup/e2e-tests/mise-tasks/test-dkg-happy-path (new) Notes for next iteration: - Could not run the full e2e suite in this sandbox (no rolling-shutter docker image available, network restrictions on mise tool installs). The task was validated by Python AST parse, e2e_utils import check, and appearance in `mise tasks` output from the e2e-tests directory. First real run will need a host with the built image. - Remaining issues: `003-test-dkg-offline-recovery` (unblocked), then `004-test-runner` (depends on 003). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements `test-dkg-offline-recovery` under `mise-test-setup/e2e-tests/`.
The task uses a 3-of-4 keyper set (NUM_KEYPERS=4, indices 0,1,2,3, threshold
3), starts the full infrastructure plus only keyper-0 and keyper-1 (chain
nodes for all 4 keypers are up), asserts a DKG failure row appears on
keyper-0's DB within 90s, brings keyper-2 online to reach the threshold,
asserts a success row within 120s, then runs test-decryption.
Decisions:
- Skip `mise run up` and call `docker compose up -d` with an explicit
service list so keyper-2 and keyper-3 are never created (clean cold-start
absence, not a stop-after-start). Infrastructure prep runs through
`patch-genesis` and `init-keyper-dbs` which chain through gen-compose,
deploy, gen-keyper-configs, init-chain-{seed,nodes}, up-{db,ethereum}.
- Reuse the shared `wait_for_dkg_failure` / `wait_for_dkg_success` helpers
from `e2e_utils.py` (success here uses the simple "any success row"
poll, since the recovery target is the same keyper set already on-chain).
- Env overrides are set in the script before any `mise run` call, leaving
`mise-test-setup/mise.toml` defaults unchanged.
Files:
- mise-test-setup/e2e-tests/mise-tasks/test-dkg-offline-recovery (new)
Notes for next iteration:
- Could not execute the test end-to-end in this sandbox: building the
rolling-shutter image needs to fetch postgres/sqlc archives that are
blocked by the network policy. The script was syntax-validated and
registered correctly under `mise tasks`. Local/CI run still required to
confirm timings (90s/120s) are comfortable.
- Next blocking issue is `004-test-runner.md` (the top-level `test`
task), which is now unblocked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the top-level `test` mise task that runs `test-dkg-happy-path` then `test-dkg-offline-recovery` via `#MISE depends`, providing a single entry point for CI and local "does everything still work" runs. Key decisions: - Used `#MISE depends` per the issue spec, but mise's default parallelism (jobs=4) would have let both tests `mise run clean` simultaneously and clobber each other's docker state. Added `[settings] jobs = 1` in e2e-tests/mise.toml to force serial dependency execution and fail-fast (offline-recovery is skipped if happy-path fails). Setting is scoped to the e2e-tests directory. - Task body is a single echo: the real work happens entirely in the depends; the body just confirms the suite finished. Files changed: - mise-test-setup/e2e-tests/mise-tasks/test (new) - mise-test-setup/e2e-tests/mise.toml (added [settings] jobs = 1) Verification: dry-run shows the correct execution order (happy-path → offline-recovery → test). Fail-fast and exit-code behaviour were confirmed against a synthetic mise project (t1 exit 1 => t2 skipped, parent exits 1; both pass => parent exits 0). The actual e2e suite was not executed end-to-end in this environment because the docker stack and tool installations (postgres, sqlc) are not reachable from the sandbox; running `mise run test` on a developer machine remains the integration check. Notes for next iteration: the `jobs = 1` setting also affects any other mise invocation made from `e2e-tests/`. All existing parent tasks are already sequential by construction, so there is no expected regression, but worth keeping in mind if future parallel tasks are added in this subtree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a keyper set excludes keyper-0 (e.g. the {3,4,5} zero-overlap
transition in the happy-path test), keyper-0 never writes a dkg_result
or decryption_key row for that eon. Hardcoding index 0 caused wait-for-dkg
to loop forever (hitting the 120s timeout) and would have caused the same
stall in wait-for-decryption-key.
Fix: add --keyper-index <N> (default 0) to wait-for-dkg,
wait-for-decryption-key, and test-decryption, and thread it through
e2e_utils.wait_for_dkg_success and add_set_and_verify in the happy-path
test. The eon and identity_registered_event lookups stay on keyper-0,
which is safe: smstate.go InsertEon runs before the FindAddressIndex
membership check, so all keypers record every eon regardless of set
membership. Only the dkg_result and decryption_key queries need to target
a member of the set.
The happy-path test passes keyper_index=3 for the {3,4,5} set; all other
sets include keyper-0 so the default suffices.
…stry The new DKGContract and ECIESKeyRegistry contracts live in the top-level foundry-based contracts repo at ../contracts/src/common/ and use pragma ^0.8.22 with OpenZeppelin v5. The existing hardhat abigen pipeline is pinned to solc 0.8.9 with OZ v4 and cannot compile them; bumping it would require non-trivial multi-compiler + OZ v5 plumbing. Instead, add a parallel foundry-based binding script that: - runs `forge build` in the top-level contracts repo - builds a synthetic combined-json from the DKGContract and ECIESKeyRegistry forge artifacts - runs `abigen --combined-json` to emit a single contract/binding_dkg.abigen.gen.go in the same `contract` package as the existing hardhat-generated bindings Wired into `go generate ./contract` (and therefore `make abigen`) via a second //go:generate directive in contract/doc.go, alongside the existing hardhat pipeline. Subsequent issues (002, 003, 005) consume DKGContract and ECIESKeyRegistry from this package. Files changed: - contracts/scripts/abigen_dkg.sh (new): forge build + abigen pipeline - rolling-shutter/contract/doc.go: add //go:generate for the new script - rolling-shutter/contract/binding_dkg.abigen.gen.go (generated) Notes for next iteration: - The script depends on the top-level ../contracts/ foundry repo being present and on `forge` being on PATH. Once that repo is published as a Go module, the binding generation could move to vendored upstream bindings instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the contract-based ECIES key registry path that replaces the Shuttermint-driven encryption-key broadcast. New work: - chainsync: ECIESKeySyncer follows the KeyperSetSyncer pattern — initial poll across all keyper sets (via getKey()) plus a live subscription to KeyRegistered. New event type, handler type, WithSyncECIESKey / WithECIESKeyRegistry options, and an ECIESKeyRegistry field on the chainsync Client. - DB: new ecies_keys table via V3_ecies_keys migration plus Upsert/Get/Exists queries (regenerated sqlc). - gnosis keyper: handler upserts every observed key into ecies_keys; processNewKeyperSet now also submits ECIESKeyRegistry.registerKey when the keyper is a member and no row exists in the local cache — initial-poll backfill is the source of truth for "already registered?" so a restart after a previous registration is silent. - config: GnosisContractsConfig gains ECIESKeyRegistry (required). - mise-test-setup: require ECIESKeyRegistry to be deployed and emit it under [Gnosis.Contracts]; corrected the gnosis section name in gen-keyper-configs so the Sequencer/ValidatorRegistry/ECIESKeyRegistry addresses actually land where the Go config expects them. Decisions: - Initial poll dedupes keypers across sets and skips addresses where getKey() returns empty, so the syncer never delivers a spurious zero-length key. - The "have I registered?" check is a single ExistsECIESKey lookup on own address; no separate sent-flag column. Registration uses the existing Shuttermint EncryptionKey (the same secp256k1 ECIES key already in the keyper config). That config field stays for now and is removed in issue 006. - ecies_keys lives in the shared keyper DB (per PRD), not the gnosis subschema, so a future service-side path can read the same rows. Not run here: the docker-based e2e tests in mise-test-setup/e2e-tests — mise refuses to install the pinned sqlc/postgres in this sandbox (403s on downloads.sqlc.dev and ftp.postgresql.org). Go build, go vet, and the existing gnosis unit tests all pass. Next: issue 003 (DKG message syncer) and issue 005, which will read ecies_keys when encrypting PolyEvals to other keypers.
…ions On each new Gnosis Chain block, the gnosis keyper now computes the current DKG phase for every active DKG Instance from the block number and the on-chain constants `PHASE_LENGTH` / `DKG_LEAD_LENGTH`, and submits the appropriate DKG Contract transaction when a phase boundary crosses. Idempotency is by querying our own message row in the per-type tables; the retry counter is derived from block arithmetic (never stored). In-memory `puredkg` state is rebuilt from the stored messages on demand (per ADR 0003): the polynomial is sampled at `StartPhase1Dealing` and lost on restart, so action handlers gate on the in-memory phase matching the expected previous value and skip submitting otherwise. The retry continues naturally on the next cycle with a fresh polynomial. Also lands the leftover scaffolding for issue 003 (DKG Contract event syncer, per-message DB tables, eons-row insertion on KeyperSetAdded) which were documented in done/ but not committed. Key files added/changed: - keyperimpl/gnosis/dkgphase.go(+_test): pure phase arithmetic - keyperimpl/gnosis/dkgmanager.go(+_test): in-memory puredkg lifecycle and replay from DB - keyperimpl/gnosis/dkgrun.go: phase-boundary dispatch (Dealing, Accusing, Apologizing, Finalizing) with on-chain hasVoted gating and dkg_result failure recording on retry rollover - keyperimpl/gnosis/newdkgevent.go: apply incoming events to the in-memory instance, in addition to storing them - keyperimpl/gnosis/newblock.go: call processDKGBlock after the validator/sequencer syncers - keyperimpl/gnosis/keyper.go: load PHASE_LENGTH/DKG_LEAD_LENGTH from the contract once at startup; wire dkgManager into Keyper struct - medley/chainsync/syncer/dkg.go + options/client/events/handler: the DKG event subscription scaffolding (issue 003) - keyper/database/sql/migrations/V4_dkg_messages.sql + generated sqlc query/model code for the new message tables Notes / blockers for next iteration: - e2e tests in mise-test-setup/e2e-tests could not run in this sandbox: the bundled anvil container is linux/amd64 and fails `exec format error` on the linux/arm64/v8 host. Go unit tests pass. The e2e tests also default to DEPLOYMENT_TYPE=service which exercises the legacy shuttermint DKG path, not the new gnosis code added here — running the suite with DEPLOYMENT_TYPE=gnosis is needed to validate the new flow end-to-end. - Issue 006 (Shuttermint removal) is the next step and will delete the redundant DKG flow that still lives in keyper/smobserver/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ract DKG
Following the contract-based DKG implementation in issues 001-005, this
commit removes all Shuttermint-specific code, tables, and configuration
from the Go keyper. The keyper no longer connects to a Tendermint chain;
DKG happens entirely through the on-chain DKGContract and ECIESKeyRegistry.
Key decisions
* Tables removed via a new V5 migration (DROP TABLE IF EXISTS so it works
on fresh installs after the legacy schema CREATEs them once):
tendermint_batch_config, tendermint_encryption_key,
tendermint_outgoing_messages, tendermint_sync_meta, poly_evals, puredkg,
outgoing_eon_keys, last_batch_config_sent, last_block_seen. Legacy
schema retained as-is so we don't have to make older migrations
idempotent.
* The keyper's ECIES private key, previously hidden inside
ShuttermintConfig.EncryptionKey, is now a required top-level field on
the gnosis Config (ECIESPrivateKey). Non-gnosis keyper impls
(shutterservice, primev, optimism, snapshot) no longer carry it; they
don't participate in DKG. That matches the PRD's scope.
* Database.GetKeyperIndex rewritten to read keyper membership from the
chainobserver keyper_set table (eon == keyper_config_index) rather than
the dropped tendermint_batch_config.
* The (now unused) Shuttermint chain command, bootstrap CLIs, app/ ABCI
application, shmsg protobufs, optimism/bootstrap pkg, and sandbox
testclient are all deleted — they only existed to talk to Shuttermint.
* go mod tidy removes tendermint and friends.
* e2e harness updated: chain-seed / chain-N docker services and the
init-chain-* / patch-genesis mise tasks are gone; gen-keyper-configs
no longer writes a [Shuttermint] section.
Files changed (groups)
* rolling-shutter/keyper/** - keyper.go / options.go / extend.go /
keypermetrics rewritten; smobserver/, fx/, shutterevents/, dkgphase/,
eonpkhandler.go, keyper_test.go deleted
* rolling-shutter/keyper/database/sql/** - V5 migration, trimmed
queries, regenerated sqlc code
* rolling-shutter/keyper/kprconfig/config.go - ShuttermintConfig removed
* rolling-shutter/keyperimpl/{gnosis,shutterservice,primev,optimism,snapshot}/**
* rolling-shutter/{app,shmsg,eonkeypublisher,sandbox/testclient}/** - deleted
* rolling-shutter/cmd/{chain,bootstrap}/** - deleted
* rolling-shutter/cmd/optimism/bootstrap.go - deleted
* rolling-shutter/keyperimpl/optimism/bootstrap/** - deleted
* rolling-shutter/medley/testsetup/eon.go - uses chainobserver
InsertKeyperSet instead of dropped InsertBatchConfig
* mise-test-setup/** - dropped chain-seed services and SHUTTERMINT_* env
vars
* rolling-shutter/go.mod / go.sum - tidied (tendermint removed)
Blockers / notes for next iteration
* e2e test suite (test-dkg-happy-path / test-dkg-offline-recovery) was
NOT exercised:
(1) sandbox can't install postgres@14.2 or sqlc binary (HTTP 403),
(2) per issue 005 the anvil image is amd64 and the host is arm64,
(3) Deploy.{service,gnosh}.s.sol still don't deploy DKGContract, so
a running keyper would fail at loadDKGContractParams,
(4) shutterservicekeyper has no contract-DKG implementation, so the
default DEPLOYMENT_TYPE=service still can't drive DKG end-to-end.
All four are out of scope per the PRD ("CI pipeline integration for
the e2e tests" / "deployment types other than service") but are the
reason the e2e acceptance checkbox in 006-remove-shuttermint.md is
unchecked.
* go build ./..., go vet ./..., go test -short ./... all pass locally.
The DKGContract now accepts and emits per-receiver evaluations as a native bytes[], so abigen handles encoding and decoding. The bespoke EncodePolyEvalBlob/DecodePolyEvalBlob helpers are no longer needed. Key decisions: - ReceiverIndicesForSender is preserved (still used by dkgrun.go and newdkgevent.go) and moved to dkgreceivers.go along with its test; the bespoke ABI-encoding helpers are deleted outright. - DKGEvent.PolyEval (bytes) renamed to PolyEvals ([][]byte) to mirror the Solidity field name and the regenerated binding struct. Files changed: - rolling-shutter/contract/binding_dkg.abigen.gen.go: regenerated against the new ABI; SubmitDealing takes [][]byte; DKGContractDealingSubmitted.PolyEvals is [][]byte - rolling-shutter/keyperimpl/gnosis/dkgrun.go: drop EncodePolyEvalBlob, pass encryptedEvals slice directly into SubmitDealing - rolling-shutter/keyperimpl/gnosis/newdkgevent.go: read ev.PolyEvals directly (no DecodePolyEvalBlob) - rolling-shutter/keyperimpl/gnosis/dkgreceivers.go (new): hosts the preserved ReceiverIndicesForSender helper - rolling-shutter/keyperimpl/gnosis/dkgreceivers_test.go (renamed from dkgpolyeval_test.go): tests for ReceiverIndicesForSender only - rolling-shutter/keyperimpl/gnosis/dkgpolyeval.go: deleted - rolling-shutter/medley/chainsync/event/events.go: DKGEvent.PolyEval -> PolyEvals ([][]byte) - rolling-shutter/medley/chainsync/syncer/dkg.go: forward ev.PolyEvals from the binding struct unchanged Blockers / notes for next iteration: - e2e tests were not exercised: the foundry/anvil image is linux/amd64 only, and this sandbox is aarch64, so the ethereum container exits with "exec format error". Forge tests (113 pass), go build ./..., go vet, and short unit tests all pass.
Add `dkg_contract`, `phase_length`, and `lead_length` columns to the core `eons` table (migration V6). When the gnosis keyper processes a `KeyperSetAdded` event for a set it belongs to, it now reads the keyper set's deployed contract address from the new `event.KeyperSet.Contract` field, calls `Keyperset.GetDKGContract()` to learn the governing DKG contract, then reads `PHASE_LENGTH` / `DKG_LEAD_LENGTH` from that contract and persists all three on the eons row alongside the existing activation block / keyper-config-index. Key decisions: - Columns are nullable. Migrated databases can roll forward without a backfill; `processDKGBlock` falls back to the startup-time values loaded from the config-supplied DKG contract (with a warning log) when columns are NULL. - Failures during the contract lookup (RPC error, missing method on an old keyper set, zero DKG address) are logged and surfaced as NULL columns so a flaky RPC never blocks joining a keyper set. - Use a `replace github.com/shutter-network/contracts/v2 => ../../contracts` directive so the regenerated KeyperSet binding (which exposes `GetDKGContract`) is picked up. The upstream tag has not been published yet. - `event.KeyperSet` grew a `Contract` field so the syncer can hand the per-keyper-set contract address to handlers without an extra `GetKeyperSetAddress` round trip. - Shutterservice does not insert eons rows in production and has no DKGContract config, so the handler there is unchanged; the shutterservice acceptance criterion is parked until that keyper grows its own DKG flow. Files changed: - rolling-shutter/keyper/database/sql/migrations/V6_eons_per_keyperset_phase_params.sql (new) - rolling-shutter/keyper/database/sql/queries/keyper.sql - rolling-shutter/keyper/database/keyper.sqlc.gen.go (regenerated) - rolling-shutter/keyper/database/models.sqlc.gen.go (regenerated) - rolling-shutter/keyperimpl/gnosis/newkeyperset.go - rolling-shutter/keyperimpl/gnosis/dkgrun.go - rolling-shutter/medley/chainsync/event/events.go - rolling-shutter/medley/chainsync/syncer/keyperset.go - rolling-shutter/go.mod, go.sum Verification: `go build ./...`, `go vet ./...`, and `go test -short ./...` pass. The e2e tests in `mise-test-setup/e2e-tests` could not be exercised in this sandbox (Foundry container is wrong-arch, and mise cannot install the postgres / sqlc tools the harness pulls down). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The happy-path test extends the keyper set to indices 3,4,5 in its
zero-overlap replacement step, which requires keyper-4 and keyper-5
configs, databases, and compose services. The parent
mise-test-setup/mise.toml defaults NUM_KEYPERS to 4 (resolved via
get_env at task evaluation time), and a #MISE env={NUM_KEYPERS = "6"}
on the test script's header does not survive the script's own
sub-`mise run` invocations of clean/gen-compose/etc., which re-evaluate
the parent config. Adding an [env] block to the child mise.toml in
e2e-tests/ makes mise's directory-walking discovery apply
NUM_KEYPERS=6 to every `mise run` initiated from that directory,
including the child invocations spawned by the test scripts.
Verified:
- `mise env` from e2e-tests/ exports NUM_KEYPERS=6
- `mise env` from mise-test-setup/ still exports NUM_KEYPERS=4 (parent
default preserved, per acceptance criterion)
- `mise run gen-compose` from e2e-tests/ generates compose.keypers.yml
with services keyper-0 .. keyper-5
Files changed:
- mise-test-setup/e2e-tests/mise.toml: added [env] block setting
NUM_KEYPERS = "6"
Notes for next iteration:
- A full `mise run test-dkg-happy-path` could not be run end-to-end
in this sandbox because mise cannot fetch the postgres/sqlc tool
binaries (network 403) and the foundry/anvil image is amd64-only on
this arm64 host. The env-propagation fix itself is verified above;
the full happy-path needs to be re-run on a normal dev environment
to confirm acceptance criterion 1 ("reaches the 3,4,5 transition
without FileNotFoundError on keyper-4.toml").
- The issue's acceptance criterion mentions docker compose services
`chain-4`, `chain-5`; no `chain-N` services exist in any compose
template in this repo, so only the keyper-N criterion was verifiable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Match the contract's `dkgStart` function name so the Go helper and the on-chain `DKGContract.dkgStart` can be cross-referenced without confusion. Pure rename — no behavioural change. Affects: - rolling-shutter/keyperimpl/gnosis/dkgphase.go (function + doc comments) - rolling-shutter/keyperimpl/gnosis/dkgphase_test.go (call sites) Unit tests for the phase-arithmetic package pass. e2e tests not run here (sandbox cannot install postgres/sqlc via mise), but this is a name-only refactor and the touched function has no other callers. Closes docs/dkg-module-refactor/dkg-module/000-rename-dkgstart.md (moved to done/ in parent repo). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a generic database-backed transaction outbox so that producers
(currently DKG; ECIES registration follows) can commit transaction intent
atomically with their business state and have a separate service handle
signing, submission, and receipt tracking.
Key decisions:
- Migration ships as V7_tx_outbox.sql; V6 was already taken by the
per-keyper-set DKG phase params work.
- Status lifecycle pending -> submitted -> confirmed | failed with a
(status, id) index so the poll loops avoid full-table scans.
- TxSender depends on a narrow Client interface (chainid, nonce, gas,
send, receipt) and is fed chainsync.Client in production.
- Submission failures that are clearly terminal (estimate-gas revert,
signing, send) mark the row failed; transient RPC errors leave it
pending for the next tick. Receipts that arrive with a non-success
status mark the row failed.
- EnqueueTx(ctx, tx, to, data, value) is exposed for producers to call
from inside their own DB tx; DKG callsites in dkgrun.go still use the
binding's Transact and will be switched in the outbox-wiring issue.
Files changed:
- rolling-shutter/keyper/database/sql/migrations/V7_tx_outbox.sql (new)
- rolling-shutter/keyper/database/sql/queries/keyper.sql (5 outbox queries)
- rolling-shutter/keyper/database/{keyper,models}.sqlc.gen.go (regen)
- rolling-shutter/txsender/{txsender.go,txsender_test.go} (new)
- rolling-shutter/keyperimpl/gnosis/keyper.go (start TxSender)
- rolling-shutter/keyperimpl/shutterservice/keyper.go (start TxSender)
Notes for next iteration:
- The PRD's "TxSender picks up a manually-inserted pending row, submits
on-chain, marks confirmed" acceptance criterion was not exercised here:
the e2e harness in mise-test-setup/e2e-tests needs sqlc/postgres
installs that the sandbox network policy blocks (HTTP 403). Validation
of the end-to-end submit-and-confirm flow falls to issue
dkg-module-refactor/dkg-module/004-outbox-wiring-in-gnosis, which
routes real DKG transactions through TxSender against anvil.
- Out of scope (deferred): nonce-replacement on transient submit errors;
EIP-1559 fee fields (currently sends LegacyTx with SuggestGasPrice).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the KeyperSetManager dependency from ECIESKeySyncer. The initial sync now walks the registry's own keyper list via getKeyperCount / getKeyperAt, which also covers keypers who registered outside an active keyper set. Key decisions: - Remove KeyperSetManager field and the per-set membership lookup; the registry's keyper list is the authoritative source so the prior cross-set dedupe map is no longer needed. - The chainsync Client keeps its top-level KeyperSetManager binding — other syncers (keyperset, shutterstate, eonpubkey, dkg) still need it. Files changed: - rolling-shutter/medley/chainsync/syncer/ecies.go - rolling-shutter/medley/chainsync/options.go Blockers / notes: - E2E suite (mise-test-setup/e2e-tests) could not be run in this environment because mise's postgres@14.2 plugin install is blocked by the network policy (HTTPS 403 from ftp.postgresql.org). Build and go vet pass cleanly; no syncer unit tests exist. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the fat-union event.DKGEvent struct (Kind discriminant + all possible fields) with a Go interface and five concrete per-event types: DealingEvent, AccusationEvent, ApologyEvent, SuccessVoteEvent, SuccessEvent. Each implements isDKGEvent(). The DKGEventKind constants and the union struct are deleted. Consumers type-switch on the concrete pointer types. Key decisions: - Use pointer receivers on the marker method and pointer concrete types on the channel so we keep the heap-allocated event identity expected by existing code (Raw block-number, etc.). - DealingEvent.PolyEvals remains [][]byte, matching the regenerated DKG binding (issue 001 in contract-updates). - Added a small dkgEventKeys helper in newdkgevent.go to extract (keyperSetIndex, retryCounter) common to every variant; this keeps the early "already succeeded" short-circuit branch from duplicating five times. Files changed: - rolling-shutter/medley/chainsync/event/events.go: interface + concrete types, deleted DKGEventKind constants - rolling-shutter/medley/chainsync/event/handler.go: handler takes DKGEvent (the interface) - rolling-shutter/medley/chainsync/syncer/dkg.go: emit concrete variants instead of one tagged struct; deliver() now takes interface - rolling-shutter/keyperimpl/gnosis/keyper.go: channel now chan syncevent.DKGEvent, channelNewDKGEvent updated - rolling-shutter/keyperimpl/gnosis/newdkgevent.go: processNewDKGEvent type-switches; store/apply helpers take concrete event types Verification: - go build ./... clean - go test ./... passes (chainsync, keyperimpl/gnosis, keyperimpl/ shutterservice and everything else green) - e2e tests not run: this sandbox cannot install postgres@14.2 (the PostgreSQL download URL is blocked by network policy and there is no preinstalled binary). Next iteration in an environment with postgres available should run mise-test-setup/e2e-tests/test before proceeding. Closes docs/dkg-module-refactor/dkg-module/002-dkgevent-typed-interface.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each start* function now calls a new `buildPureDKG(ctx, tx, eon, retry)` that reconstructs the puredkg state from stored message rows. No in-memory cache, no mutex, no `dkgInstance` wrapper. The DB is the only source of truth for participation state. Phase-boundary triggering is unchanged. Key decisions: - Reuse the existing replay helper (now `replayStoredMessages`) but pass the puredkg pointer in directly instead of through a struct, so it's clear the data flows DB → puredkg per call. - Drop the `applyDealing/Accusation/Apology ToInstance` cache-sync helpers from `processNewDKGEvent`: event handlers now just write rows; the next `start*` invocation rebuilds from those rows. Files changed: - keyperimpl/gnosis/dkgmanager.go: replaced dkgInstance/dkgManager and loadOrBuildInstance with buildPureDKG; kept decrypt/encrypt helpers - keyperimpl/gnosis/dkgrun.go: all four start* functions call buildPureDKG - keyperimpl/gnosis/newdkgevent.go: removed apply*ToInstance helpers - keyperimpl/gnosis/keyper.go: dropped dkgManager field and constructor wiring Notes for next iteration: outbox wiring (issue 004) replaces the direct Transact calls in start* with tx_outbox writes. E2E tests not run in this sandbox (postgres 14.2 download blocked by network policy); unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each DKG start function (Dealing, Accusing, Apologizing, Finalizing) now ABI-packs its calldata via contract.DKGContractMetaData and writes a pending row to tx_outbox in the same DB transaction as its message rows; TxSender drains those rows. The target DKG contract address is resolved per eon (eons.dkg_contract column, with the config-supplied DKGContract address as fallback) by a new dkgContractAddrForEon helper called from handlePhaseBoundary and passed through to each start*. makeTransactOpts and bind.NewKeyedTransactorWithChainID are removed from the DKG path; the signing key now reaches the chain only through TxSender (via the existing Config.PrivateKey wiring in Keyper.Start). The phase-boundary trigger is unchanged at this stage — the per-block reactor lands in 005-extract-dkg-package. maybeRegisterECIESKey and hasVotedOnChain are intentionally untouched: the former is rewritten when the ECIES path moves into the new dkg module in the next issue, the latter is a read-only call. Files changed: - keyperimpl/gnosis/dkgrun.go Validation: - go build ./... and go vet ./... pass - keyperimpl/gnosis unit tests run green - e2e harness in mise-test-setup/e2e-tests cannot run inside the sandbox (network policy blocks postgres-14.2 and sqlc-1.28.0 downloads, HTTP 403 on ftp.postgresql.org and downloads.sqlc.dev — same blocker noted on the tx-outbox/000 issue). End-to-end submit-and-confirm of these outbox rows needs to be exercised on a host with network access; the next-up issue (005-extract-dkg-package) is the natural place. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lift all DKG participation logic out of `keyperimpl/gnosis/` into a new
`rolling-shutter/dkg/` package and switch from phase-boundary triggering
to a per-block reactor. The package is a pure DB-driven reactor (per
ADR 0004): the host keyper calls `Manager.HandleBlock` on every new
block and the manager iterates all active eons and dispatches the four
DKG phase actions plus ECIES key registration. Each action is idempotent
via DB-state checks ("own message row exists" / `dkg_result` row
existence / `ecies_keys` row existence).
Key decisions:
- `Manager.HandleBlock` fires all five actions unconditionally on every
block; the previous phase-boundary trigger inside `processDKGBlock`
is gone. Idempotency moves entirely to DB checks.
- `maybeRegisterECIESKey` is removed from `processNewKeyperSet` and
reimplemented in the dkg module against `tx_outbox` — the module
makes no direct chain calls.
- `hasVotedOnChain` is replaced with a local check on
`dkg_result.success = true`. The DKG module never reads the chain.
- `startDealing` now also persists a self-eval row
`(sender = ownIndex, receiver = ownIndex)` to `dkg_poly_eval`. The
rebuild path needs `pure.Evals[ownIndex]` for `ComputeResult` and
this is the minimal change that avoids a polynomial-loss schema
migration in this issue.
- `startFinalizing` bypasses puredkg's phase machinery by setting
`pure.Phase = puredkg.Finalized` directly (the field is exported);
`ComputeResult` then operates on the replayed
Commitments/Evals/Accusations/Apologies state.
Files changed:
- new rolling-shutter/dkg/{phase,puredkg,manager,dealing,accusing,
apologizing,finalizing,ecies}.go and tests
- deleted rolling-shutter/keyperimpl/gnosis/{dkgmanager,dkgphase,
dkgreceivers,dkgrun}.go and their tests
- rolling-shutter/keyperimpl/gnosis/keyper.go: drop
dkgPhaseLength/dkgLeadLength fields and `loadDKGContractParams`;
construct `dkg.Manager` in `Start`
- rolling-shutter/keyperimpl/gnosis/newblock.go: call
`kpr.dkgMgr.HandleBlock` in place of `processDKGBlock`
- rolling-shutter/keyperimpl/gnosis/newdkgevent.go: switch to
`dkg.ReceiverIndicesForSender`
- rolling-shutter/keyperimpl/gnosis/newkeyperset.go: remove
`maybeRegisterECIESKey`
Notes for next iteration:
- `go build ./...` and `go vet ./...` pass; full unit test suite
passes including the new finalize-after-replay test that proves
the recovered eon public key equals the live path.
- e2e harness in `mise-test-setup/e2e-tests` cannot run inside the
sandbox: `mise run test-dkg-happy-path` aborts at tool install
with HTTP 403 on `downloads.sqlc.dev` and `ftp.postgresql.org`
(same blocker as `tx-outbox/done/000` and `dkg-module/done/004`).
End-to-end validation has to happen on a host with network
access.
- Polynomial-loss on the apology path is a known existing limitation
(ADR 0003): if accusations against us arrive, we cannot apologize
without a persisted polynomial. Happy-path and offline-recovery
e2e tests do not exercise this; if a future failure mode requires
it, a `dkg_own_polynomial` migration is the natural follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires the shutterservice keyper into the shared dkg.Manager so the per-block reactor drives DKG participation. KeyperSetAdded and DKG/ECIES event handlers now only persist raw event data; all DKG decisions happen in the dkg package. Key decisions: - Mirror the gnosis pattern verbatim — eon row eagerly inserted on KeyperSetAdded when a member, then dkg.Manager.HandleBlock fires every block. Idempotency via DB checks makes repeated calls safe. - Per-keyper-set DKG contract resolved via the KeyperSet contract's getDKGContract; falls back to the manager-configured global address when unset (NULL phase params just skip that eon, no global lookup). - ECIES private key promoted to a required config field on the shutterservice Config (was previously gnosis-only). Needed to decrypt PolyEvals. - Service deployment now also deploys DKGContract and ECIESKeyRegistry so the e2e tests can exercise the full DKG cycle against the shutterservice keyper. AddKeyperSet scripts accept an optional DKG_CONTRACT_ADDRESS env to wire each new keyper set's DKG contract. Files changed: - rolling-shutter/keyperimpl/shutterservice/keyper.go: chainsync wired with ECIES/DKG handlers; dkg.Manager constructed; processNewDKGEvent fan-out; newDKGEvents channel. - rolling-shutter/keyperimpl/shutterservice/config.go: ECIESPrivateKey field; ECIESKeyRegistry + DKGContract on ContractsConfig. - rolling-shutter/keyperimpl/shutterservice/newblock.go: HandleBlock call. - rolling-shutter/keyperimpl/shutterservice/newkeyperset.go: eon row insert with per-keyper-set DKG contract lookup (matches gnosis). - rolling-shutter/keyperimpl/shutterservice/newdkgevent.go: new — raw row-insert handlers for the five DKG event types. - rolling-shutter/keyperimpl/shutterservice/newecieskey.go: new — ECIES key cache upsert. - mise-test-setup/mise-tasks/deploy: REQUIRED_CONTRACTS now lists DKGContract + ECIESKeyRegistry for service deployment. - mise-test-setup/mise-tasks/gen-keyper-configs: writes the new addresses into the service section of keyper TOMLs. - mise-test-setup/mise-tasks/add-keyper-set: passes DKG_CONTRACT_ADDRESS to the AddKeyperSet script. Notes for next iteration: - e2e tests were not run in the sandbox: the Anvil image (ghcr.io/foundry-rs/foundry) is linux/amd64 only and crashes with "exec format error" on the arm64 sandbox host. Go build + vet pass; dkg and shutterservice unit tests pass. - The contracts container image is built from https://github.com/shutter-network/contracts.git#docker, so the Deploy.service.s.sol and AddKeyperSet*.s.sol edits in the sibling contracts/ repo will only take effect once that docker branch is updated (or the compose file is repointed at a local build context). - 007-gnosis-integration remains: clean up any residual DKG code in the gnosis keyper after the shutterservice integration is validated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tterservice Audit for issue 007 (gnosis-integration) confirmed the gnosis keyper was already fully integrated with the new dkg.Manager in issue 005: - no residual dkgManager, dkgPhaseLength, dkgLeadLength, loadDKGContractParams, dkgInstance, phase-boundary trigger code, or direct start* calls in keyperimpl/gnosis/ - KeyperSetAdded handler does DB inserts only (mirrors shutterservice) - DKG event handlers do row inserts only - processNewBlock calls dkgMgr.HandleBlock once per block Only cleanup needed: drop an unused intermediate variable in the eon-exists check in processNewKeyperSet so the form matches shutterservice's idiomatic if/else if pattern. No behavior change. Verification: - go build ./... passes - go test ./dkg/... ./keyperimpl/... passes Files: rolling-shutter/keyperimpl/gnosis/newkeyperset.go e2e tests not run in sandbox: ghcr.io/foundry-rs/foundry:v1.5.0 Anvil image is linux/amd64 only; arm64 sandbox host returns exec format error. Same blocker as issue 006. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… loop TxSender.Start used to launch two independent goroutines: one driving the submit phase and one driving the confirm phase. They never shared mutable state, but the parallel scheduling complicated reasoning and was a silent trap for future changes. Collapse both into a single pollLoop that runs the two phases sequentially on each tick: processPending first, processSubmitted second. Key decisions - Extract a poll() helper that runs both phases in order; pollLoop is now a thin ticker driver. The helper is private but lets the test exercise one iteration directly without time.Ticker timing assumptions. - Failure of either phase logs at error and continues to the next phase; matches the prior behavior of each independent loop tolerating one-tick failures. Files changed - rolling-shutter/txsender/txsender.go: replace submitLoop/confirmLoop with pollLoop + poll. Start now launches exactly one goroutine. - rolling-shutter/txsender/txsender_test.go: add fakeClient and TestPollProcessesBothPhasesInOneIteration, which seeds one pending and one submitted row and asserts both phases run within one poll(). Verification - go test ./txsender/ passes (unit and integration tests, with postgres test DB). - go build ./... passes. Blockers / notes for next iteration - e2e tests in mise-test-setup/e2e-tests could not be run in this sandbox: mise's postgres@14.2 and sqlc@1.28.0 installs fail with HTTP 403 from the upstream mirrors (network policy). Verify on a host with full network access before merging. - Next tx_sender refactor (issue 002) reorders submitRow to mark the row `submitted` before SendTransaction is called; this commit leaves the current ordering unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reorder submitRow so MarkTxSubmitted runs before SendTransaction. A
crash between the two steps now leaves the row as `submitted` with its
tx hash persisted (preserving it for future re-broadcast logic) rather
than as `pending`, which on restart would have prompted a duplicate
broadcast under a fresh nonce.
Status transitions:
- Pre-MarkTxSubmitted error: row stays `pending`, send is skipped.
- Post-MarkTxSubmitted send error: row moves `submitted -> failed`
(MarkTxFailed has no precondition on current status).
Files:
- rolling-shutter/txsender/txsender.go: reorder, update comments.
- rolling-shutter/txsender/txsender_test.go: add `sendErr` knob to
fakeClient; new `TestSubmitRowMarksFailedWhenSendFails` and
`TestSubmitRowSkipsSendWhenMarkSubmittedFails`.
Notes for next iteration:
- E2E suite (mise-test-setup/e2e-tests) couldn't run in this sandbox:
`downloads.sqlc.dev` and `ftp.postgresql.org` are blocked by the
default-deny network policy. Integration tests against a real
Postgres covered all acceptance criteria.
- Re-broadcast logic for `submitted` rows that never appear on chain
is still out of scope; this change is the structural prerequisite.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `label TEXT NOT NULL DEFAULT ''` column to tx_outbox via a new V8
migration. EnqueueTx now accepts a `label string` parameter (last
position) that callers use to describe the transaction (e.g.
"submitDealing ksi=7 retry=2"); TxSender treats it opaquely and
includes it in every log line that already carries the row ID
(submission, mark-submitted error, send failure, receipt warnings,
mark-confirmed error/success, and markFailed). A new
"tx outbox: confirmed" info log captures the success path so operators
see the full lifecycle with the label.
All five DKG/ECIES enqueue call sites (submitAccusation, submitApology,
submitDealing, submitSuccessVote, registerKey) now pass a non-empty
label formatted with ksi (keyper set index) and retry counter where
applicable, per PRD guidance to use ksi rather than eon in label text.
Files changed:
- rolling-shutter/keyper/database/sql/migrations/V8_tx_outbox_label.sql (new)
- rolling-shutter/keyper/database/sql/queries/keyper.sql (InsertPendingTx)
- rolling-shutter/keyper/database/{keyper,models}.sqlc.gen.go (regenerated)
- rolling-shutter/txsender/txsender.go (EnqueueTx signature, log fields,
markFailed signature, new confirmed log)
- rolling-shutter/txsender/txsender_test.go (TestEnqueueTxPersistsLabel)
- rolling-shutter/dkg/{accusing,apologizing,dealing,finalizing,ecies}.go
(pass label to EnqueueTx, add fmt import)
Verification:
- `go build ./...` clean
- `go vet ./...` clean
- All txsender tests pass against a real postgres (V8 migration runs,
label round-trips through EnqueueTx -> GetTxOutboxByID)
- dkg/keyper test packages still green in -short mode
- e2e tests not run: the ethereum container in the local mise harness
is linux/amd64 and the sandbox host is linux/arm64, so qemu emulation
is required and ethereum became unhealthy on `mise run
test-dkg-happy-path`. This is an environment limitation, not a
regression — flag for re-run on an amd64 host.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds migration V9 that renames the four DKG message tables to plural
form (dkg_poly_commitment->dkg_poly_commitments, dkg_poly_eval->
dkg_poly_evals, dkg_accusation->dkg_accusations, dkg_apology->
dkg_apologies) for naming consistency, and creates a new
dkg_initial_states table keyed on (keyper_config_index, retry_counter)
that will hold the gob-encoded puredkg.PureDKG blob immediately after
StartPhase1Dealing — written exactly once per DKG Instance to allow
accusations/apologies to be produced on subsequent blocks (including
across process restarts).
Adds SQLC queries InsertDKGInitialState and GetDKGInitialState (using
shdb.EncodePureDKG/DecodePureDKG for serialisation, no new code needed)
and updates the four existing rename-affected queries to reference the
new plural table names. Generated code regenerated with sqlc v1.28.0.
The schemas/keyper.sql base file does not list the dkg_* tables (they
are created by V4 and renamed by V9), so no schema-file rename is
needed; sqlc reads schemas+migrations together and resolves to the
plural names. Schema-version header bumped to keyper-23.
Files changed:
- rolling-shutter/keyper/database/sql/migrations/V9_dkg_initial_states_and_renames.sql (new)
- rolling-shutter/keyper/database/sql/queries/keyper.sql (4 renames + 2 new queries)
- rolling-shutter/keyper/database/{keyper,models}.sqlc.gen.go (regenerated)
Notes for next iteration:
- Issue 002 (maybeDeal persists initial state) is unblocked.
- E2E tests in mise-test-setup/e2e-tests could not run in sandbox due
to 403s downloading postgres and sqlc binaries; go build, go vet, and
go test ./... all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…dkg state After StartPhase1Dealing produces the polynomial and self-eval, the gob-encoded puredkg.PureDKG is now written to dkg_initial_states inside the same transaction as the commitment + per-receiver poly_eval rows and the submitDealing outbox entry. On subsequent invocations, presence of the initial-state row (not the commitment row) is the sole idempotency signal — once written, maybeDeal is a no-op regardless of downstream table state. This is the persistence half of the DKG HandleBlock refactor PRD. The stored blob is what later slices (003) will load to reconstruct puredkg at Phase=Dealing so accusations/apologies can actually be produced across process restarts. Decisions: - Use shdb.EncodePureDKG (already present) for serialisation. - Idempotency check is pgx.ErrNoRows on GetDKGInitialState rather than scanning the existing commitments table — matches the PRD acceptance criteria and keeps the check O(1). - Manager.HandleBlock dispatcher renamed to call maybeDeal. Files changed: - rolling-shutter/dkg/dealing.go: rename, switch idempotency check, add InsertDKGInitialState write before commitment/eval/outbox writes. - rolling-shutter/dkg/dealing_test.go: new integration tests covering first-call persistence and second-call no-op. - rolling-shutter/dkg/manager.go: rename callsite. - rolling-shutter/dkg/finalizing.go, dkg/puredkg_test.go: comment updates referencing the new name. Verification: - go build ./... clean. - go test ./dkg/... passes (TestMaybeDealPersistsInitialStateAndIsIdempotent, TestMaybeDealNoopWhenInitialStateExists, plus the unchanged puredkg/phase tests). - e2e suite (mise-test-setup/e2e-tests) was not run: this sandbox cannot download the required mise tools (postgres@14.2, sqlc@1.28.0) — 403 from upstream — so test-dkg-happy-path failed at the setup step. Worth running on a host with unrestricted network access before merge. Next iteration (003-buildpuredkg-phase-aware-fix-accuse-apologize.md): make buildPureDKG phase-aware and load from dkg_initial_states for the accuse/apologize/finalize paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…izing} to maybe{Accuse,Apologize}
buildPureDKG now takes a blockPhase parameter and returns puredkg at the
phase each maybe-function expects: fresh PureDKG (Phase=Off) for
PhaseDealing, loaded from dkg_initial_states then advanced through the
prerequisite phases for Accusing / Apologizing / Finalizing. Returns nil
for non-Dealing phases when no initial state exists, so a Keyper that
never dealt is skipped silently.
This unblocks accusations and apologies, which were dead code: the
phase guards `pure.Phase != puredkg.Dealing` / `!= Accusing` always
tripped because plain message replay leaves Phase at Off. maybeAccuse
now correctly calls StartPhase2Accusing on a Dealing puredkg, and
maybeApologize calls StartPhase3Apologizing on an Accusing one with
the polynomial alive from the loaded initial state. Both log at debug
when they produce no rows, distinguishing normal operation from the
silent failure.
startFinalizing is adapted to the new buildPureDKG signature (the full
rename + HandleBlock restructure lands in slice 004); it now loads the
initial state and replays through Apologizing before setting
Phase=Finalized and calling ComputeResult.
Encoding fix: after gob-decode of PureDKG, nil entries in `Commitments`
and `Evals` slices are revived as non-nil zero-value pointers (via the
elements' GobDecoder), which would defeat puredkg's duplicate-msg
check during the subsequent DB replay. buildPureDKG resets both slices
and re-derives the self-eval from the loaded polynomial.
Files changed:
- dkg/puredkg.go: buildPureDKG phase parameter + per-phase reconstruction;
replayStoredMessages split into replayCommitmentsAndEvals,
replayAccusations, replayApologies; gob-roundtrip fix
- dkg/accusing.go: rename startAccusing→maybeAccuse; drop dead
Phase!=Dealing guard; call StartPhase2Accusing; add debug log
- dkg/apologizing.go: rename startApologizing→maybeApologize; drop
dead Phase!=Accusing guard; call StartPhase3Apologizing; add debug log
- dkg/dealing.go: adapt maybeDeal to new (pure==nil) skip convention
- dkg/finalizing.go: adapt startFinalizing to new buildPureDKG signature
- dkg/manager.go: update HandleBlock to call renamed functions
- dkg/accusing_test.go, apologizing_test.go, build_puredkg_test.go: new
Tests verified locally against a postgres:14-alpine container; e2e
mise-test-setup is blocked in this sandbox (postgres + sqlc downloads
return 403). Next slice (004) restructures HandleBlock to gate dispatch
on the block-level phase and share a single transaction per eon.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jannikluhn
force-pushed
the
unshuttermintify
branch
from
cbdf7fd to
8d1ed93
Compare
added 5 commits
July 8, 2026 20:40
Rename V{N}_*.sql to V{NNN}_*.sql so directory listings sort naturally
and future versions have consistent width. The migration loader parses
versions with fmt.Sscanf %d, so leading zeros require no code changes.
Trivial fixes: correct British→American spelling in comments, replace string(...)==string(...) with bytes.Equal, rename unused parameters to _, break long interface method declarations, add missing PhaseNone switch cases, drop dead retryCounter param and always-nil error return. Silence per-line gosec G115 warnings for keyper/retry indices, block arithmetic, thresholds, and nonces (all domain-bounded). Add per-function gocyclo/funlen nolints where refactoring would fragment a cohesive select-dispatcher, orchestration flow, or test narrative.
Remove docs for the bootstrap, chain, and op-bootstrap commands that no longer exist, and refresh the synopsis lines that used to mention Shuttermint on gnosiskeyper, op-keyper, primevkeyper, snapshotkeyper, and shutterservicekeyper.
| Int("apology-count", apologyCount) | ||
| } | ||
| event.Msg("DKG succeeded") | ||
| return queries.InsertDKGResult(ctx, corekeyperdb.InsertDKGResultParams{ |
Contributor
There was a problem hiding this comment.
success=true is hardcoded but pureBytes can be nil (when ComputeResult fails or when pure is nil), so we store a success row with no share. Downstream treats it as a real success and the keyper can't produce shares for this eon, with no retry. Can we only store success=true+nil for the non-member case, and otherwise skip or write success=false with the error?
| if err != nil { | ||
| // Gas estimation failures usually mean the transaction would revert; | ||
| // record it as failed so an operator can inspect the calldata. | ||
| s.markFailed(ctx, row.ID, row.Label, errors.Wrap(err, "estimate gas")) |
Contributor
There was a problem hiding this comment.
good to have: the errors here can be of transient nature like rate-limits if using RPC provider or if the tx is already in mempool, we can disqualify these errors and should not mark them as failed, as they can be retried
| } else if !errors.Is(err, pgx.ErrNoRows) { | ||
| return errors.Wrap(err, "check existing eon row") | ||
| } | ||
| dkgContract, phaseLength, leadLength := kpr.fetchDKGParamsForKeyperSet(ctx, ev.Contract) |
Contributor
There was a problem hiding this comment.
fetchDKGParamsForKeyperSet is called inside the open BeginFunc tx, so up to 5 RPC round-trips run while holding a pool connection and an uncommitted write. On any RPC error it returns NULLs which InsertEon persists and GetEon skips re-insert on redelivery so it never retries. A NULL row is then fatal every block downstream, so the keyper is permanently excluded from this set's DKG. Can we move the RPC out of the tx and retry on failure instead of writing NULL?
| // HandleBlock — once per discovered keyper set, after the keyper set | ||
| // row is committed (MaybeRegisterECIESKey reads it). It is idempotent | ||
| // and a no-op for non-members. | ||
| return kpr.dkgMgr.MaybeRegisterECIESKey(ctx, keyperSetIndex) |
Contributor
There was a problem hiding this comment.
nit pick: New keypers can only register their ECIES key after joining a set (newkeyperset.go:97, ecies.go:43-47), so if (activationBlock - addedBlock) - dkgLeadLength is smaller than the registerKey tx inclusion+indexing time, maybeDeal deals empty slots to the unregistered keyper (puredkg.go:318) and its first round degrades into false accusations; can we document this timing invariant and defer dealing to a receiver whose key is not yet cached instead of dealing an empty slot?
blockchainluffy
left a comment
blockchainluffy
left a comment
Contributor
There was a problem hiding this comment.
The shuttermint code is deleted in this PR but the Primev implementation uses it still, just a thing to keep in mind that this version will not support Primev
jannikluhn
force-pushed
the
unshuttermintify
branch
from
dcd90ca to
620009f
Compare
Adds the keyper side of the DKG retry ceiling introduced by the on-chain work in ticket 01. `eons` gains a `max_retries BIGINT NOT NULL` column captured once at keyper-set registration from `DKGContract.MAX_RETRIES()`, so a future contract redeploy cannot retroactively affect a set already in progress. `PhaseAt` grows a new `maxRetries` parameter (positioned before `retryCounter` to mirror the contract signature) and returns `PhaseNone` as its first guard when `retryCounter >= maxRetries`. The manager threads the value through `phaseParamsForEon` into `PhaseAt` with no new branches — the existing `PhaseNone` early-return in `handleEon` is the single gate. A well-behaved keyper therefore stops broadcasting DKG messages at the retry ceiling. Also adds a local `replace` directive for `shutter-network/contracts/v2` so the updated bindings (including `MAXRETRIES()`) shipped in ticket 01 are consumed until a new upstream tag lands.
jannikluhn
force-pushed
the
unshuttermintify
branch
from
620009f to
39baecd
Compare
added 9 commits
July 15, 2026 15:51
EstimateGas and SendTransaction failures were unconditionally recorded as terminal, which also caused graceful shutdown (context.Canceled) to fail in-flight rows. Introduce isTransient as a positive allowlist (context errors, syscall.Errno, io.EOF/ErrUnexpectedEOF, net.Error) and a markFailedUnlessTransient wrapper applied at those two call sites; other sites that already retry on all errors are unchanged.
The function silently swallowed RPC and validation errors, returning NULL sql values and maxRetries=0. The docstring claimed downstream code would fall back to a config-supplied DKG contract, but the DKG manager actually treats NULL phase params as a fatal configuration error with no chain-client fallback (rolling-shutter/dkg/manager.go phaseParamsForEon). The result was that any transient RPC failure permanently poisoned the eons row for that keyper set, and any real MAX_RETRIES=0 on chain was indistinguishable from a fetch failure. Return an error from every failure path in both gnosis and shutterservice copies. The caller wraps and returns it, so the outer event loop logs the failure and the eon row is not inserted; chainsync re-delivery on the next initial poll retries the fetch.
fetchDKGParamsForKeyperSet issues up to five sequential RPC round-trips to the keyper-set and DKG contracts. Doing this inside BeginFunc held the transaction (and the row locks acquired by InsertKeyperSet) open across network I/O, which is a known contention anti-pattern. Hoist the fetch above BeginFunc, gated on isMember (which is computed from the event before either the fetch or the transaction). On event replay the existing-row check inside the tx still short-circuits the insert, so at worst we spend one extra round-trip when chainsync re-delivers a KeyperSetAdded event we already processed. Applied identically to gnosis and shutterservice keyperimpls.
Replace the vague "same architectural level as HandleBlock" wording with the concrete invariant (must run after the tx commits) and note that registration can still land too late if a keyper set is added inside its own lead-time window, which only bites new members.
Extract the membership, prior-success, and retry-ceiling filters out of the per-eon dispatch into a new `activeDKGs` helper. `HandleBlock` now asks `activeDKGs` which eons this keyper must still consider at this block, then dispatches each one to the renamed `processDKG` (was `handleEon`), whose body shrinks to per-block phase math. This is the seam ticket 02 will drop the supersession filter into. No new SQL queries; existing filters preserved. PhaseAt keeps its retryCounter >= maxRetries guard as belt-and-braces.
Add the fourth "over or not mine" filter: a Keyper Set N is superseded at the current block when some later Keyper Set M (M > N) has already activated (activationBlock_M <= currentBlock). Superseded sets are omitted from activeDKGs so no further Dealing, Accusing, Apologizing, or Finalizing submissions happen, even mid-cycle. The predicate reuses the observer's existing "latest keyper set at block" query; no new SQL. HandleBlock now emits a single log.Debug summary per block with counts by filter reason (not-member, succeeded, superseded, retries-exhausted) instead of a per-skip line. HandleDKGSuccess, PhaseAt, and CurrentRetryCounter are untouched; a peer-driven success on a superseded set is still recorded via the unchanged handler. Seam 1 (unit): table-driven activeDKGs test covering all nine cases from the ticket-02 checklist. Seam 2 (integration): two Manager tests verifying no tx_outbox / dkg_sent_actions rows are emitted for the superseded older set, and that peer-driven success still writes dkg_result unchanged.
The synthetic SuccessEvent emitted for already-completed DKGs at startup now carries the retry counter reported by the contract's succeededAtRetry(ksi) view instead of a hardcoded zero. HandleDKGSuccess uses that value to rebuild puredkg state, so reporting the true retry is required to reconstruct the correct dkg_result for DKGs that finished on a later retry. Bumps contracts/v2 to the commit that exposes succeededAtRetry.
jannikluhn
force-pushed
the
unshuttermintify
branch
from
c1290b7 to
6d0dc7d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run the DKG on the Ethereum-like chain that hosts the rest of the contracts as well. Depends on shutter-network/contracts#9
It has been tested locally with the mise test setup and the added e2e tests. Not ready to merge, but putting it here for early review.