Skip to content

Instance heartbeat on the monitoring stream, and report stop() as a restartable stop - #198

Merged
vanrogu merged 2 commits into
developfrom
claude/modest-tesla-mkz3m2
Nov 1, 2026
Merged

vanrogu merged 2 commits into
developfrom
claude/modest-tesla-mkz3m2

Conversation

@vanrogu

@vanrogu vanrogu commented Nov 1, 2026

Copy link
Copy Markdown
Member

Why

A reader of the monitoring stream (the dashboard) could not tell an idle instance from a killed one. A SIGKILL'd process never reports BoundedContextStopping/Stopped, so it stayed "running" forever, even after its leases had moved to other instances. Separately, stop(), whether called in code or triggered by an operator's StopBoundedContext instruction, emitted no lifecycle event at all, so a stopped context also kept reading as running.

What

Instance heartbeat (StreamAppendingBoundedContextListener, new InstanceHeartbeat)

  • Each bounded context the listener reports on gets a heartbeat. It is a bookmark holding a read position at the monitoring stream's head, re-placed every 30s. It starts at BoundedContextStarting, and terminating the context removes it.
  • The interval is a new constructor argument; Duration.ZERO turns the heartbeat off.
  • A bookmark, not an event: there is one row per context/logical/physical name, and every placement refreshes its updatedAt. A heartbeat event would add about a million events per instance per year.
  • One virtual thread per listener places the beats, parked on a Condition between them. A beat in flight cannot put the row back after termination has removed it. A failing beat logs WARN once, and a closed store ends the beats.
  • InstanceHeartbeat defines the format: reader prefix, tags, of(Bookmark) → Beat. A reader should treat a beat more than 3 intervals old as a dead process.

stop() is reported (BoundedContextImpl, BoundedContextEvent)

  • stop() now emits BoundedContextStopping/Stopped with a new restartable = true; terminate() emits the same pair with false.
  • A flag rather than new event types, so a reader older than this change still sees the stop. A new event type would make its typed read of the stream fail.
  • Events stored before the field existed read as false (@JsonSetter(nulls = AS_EMPTY)). The existing 4-arg constructors stay and build a termination, so callers compile unchanged.
  • The heartbeat keeps beating through a restartable stop, because the process is still up.

Tests

  • InstanceHeartbeatTest (beat with tags at the head, refreshed while idle, removed on terminate with no beat placed afterwards, stop() keeps beating, zero interval, overdue rule)
  • BoundedContextStopEventsTest (stop then start, terminate, second stop, stop then terminate, an event stored without the field)
  • The full suite passes on inmem and inmem-fs. The Postgres backends were not run (no Docker where this was built). The heartbeat relies on the Postgres bookmark upsert refreshing updated_at on every placement; the SQL does that, but CI's Postgres run of InstanceHeartbeatTest is the real check.

The dashboard change that reads the heartbeat and the restartable flag is in sliceworkz/dashboard on the branch of the same name, and needs this merged and the snapshot republished first.


Generated by Claude Code

A killed process never reports that it stopped, so a reader of the
monitoring stream kept showing an idle context on a dead process as
running. StreamAppendingBoundedContextListener now keeps a heartbeat per
bounded context it reports on: a read-position bookmark at the head of
the monitoring stream, refreshed every 30s (configurable, zero disables)
from BoundedContextStarting until BoundedContextStopping, and removed on
termination. One bookmark row per deployed copy of a context, instead of
an event per beat.

InstanceHeartbeat carries the wire format (reader prefix, tags) and the
reader-side helper that turns a bookmark back into a Beat and judges it
overdue after three missed intervals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJQ5EEnnbARTP5KJwEWZLx
stop() emitted no lifecycle event, so a context taken out of service in
code or by an operator's StopBoundedContext instruction read as running
to every reader of the monitoring stream. It now emits
BoundedContextStopping and BoundedContextStopped with restartable = true;
terminate() emits the same pair with false. The flag reads false on
events stored before it existed, and the four-argument constructors
build a termination. The instance heartbeat keeps beating through a
restartable stop, since the process is still up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJQ5EEnnbARTP5KJwEWZLx
@vanrogu
vanrogu merged commit 2b11c3a into develop Nov 1, 2026
2 checks passed
@vanrogu
vanrogu deleted the claude/modest-tesla-mkz3m2 branch November 1, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants