Repository navigation
Instance heartbeat on the monitoring stream, and report stop() as a restartable stop - #198
Merged
Merged
Conversation
A killed process never reports that it stopped, so a reader of the monitoring stream kept showing an idle context on a dead process as running. StreamAppendingBoundedContextListener now keeps a heartbeat per bounded context it reports on: a read-position bookmark at the head of the monitoring stream, refreshed every 30s (configurable, zero disables) from BoundedContextStarting until BoundedContextStopping, and removed on termination. One bookmark row per deployed copy of a context, instead of an event per beat. InstanceHeartbeat carries the wire format (reader prefix, tags) and the reader-side helper that turns a bookmark back into a Beat and judges it overdue after three missed intervals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJQ5EEnnbARTP5KJwEWZLx
stop() emitted no lifecycle event, so a context taken out of service in code or by an operator's StopBoundedContext instruction read as running to every reader of the monitoring stream. It now emits BoundedContextStopping and BoundedContextStopped with restartable = true; terminate() emits the same pair with false. The flag reads false on events stored before it existed, and the four-argument constructors build a termination. The instance heartbeat keeps beating through a restartable stop, since the process is still up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PJQ5EEnnbARTP5KJwEWZLx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A reader of the monitoring stream (the dashboard) could not tell an idle instance from a killed one. A SIGKILL'd process never reports
BoundedContextStopping/Stopped, so it stayed "running" forever, even after its leases had moved to other instances. Separately,stop(), whether called in code or triggered by an operator'sStopBoundedContextinstruction, emitted no lifecycle event at all, so a stopped context also kept reading as running.What
Instance heartbeat (
StreamAppendingBoundedContextListener, newInstanceHeartbeat)BoundedContextStarting, and terminating the context removes it.Duration.ZEROturns the heartbeat off.updatedAt. A heartbeat event would add about a million events per instance per year.Conditionbetween them. A beat in flight cannot put the row back after termination has removed it. A failing beat logs WARN once, and a closed store ends the beats.InstanceHeartbeatdefines the format: reader prefix, tags,of(Bookmark)→Beat. A reader should treat a beat more than 3 intervals old as a dead process.stop()is reported (BoundedContextImpl,BoundedContextEvent)stop()now emitsBoundedContextStopping/Stoppedwith a newrestartable = true;terminate()emits the same pair withfalse.false(@JsonSetter(nulls = AS_EMPTY)). The existing 4-arg constructors stay and build a termination, so callers compile unchanged.Tests
InstanceHeartbeatTest(beat with tags at the head, refreshed while idle, removed on terminate with no beat placed afterwards,stop()keeps beating, zero interval, overdue rule)BoundedContextStopEventsTest(stop then start, terminate, second stop, stop then terminate, an event stored without the field)inmemandinmem-fs. The Postgres backends were not run (no Docker where this was built). The heartbeat relies on the Postgres bookmark upsert refreshingupdated_aton every placement; the SQL does that, but CI's Postgres run ofInstanceHeartbeatTestis the real check.The dashboard change that reads the heartbeat and the
restartableflag is in sliceworkz/dashboard on the branch of the same name, and needs this merged and the snapshot republished first.Generated by Claude Code