Skip to content

feat: add fleet energy telemetry endpoint - #63

Open
0xdfi wants to merge 3 commits into
MiaAI-Lab:mainfrom
0xdfi:feat/fleet-energy-endpoint
Open

feat: add fleet energy telemetry endpoint#63
0xdfi wants to merge 3 commits into
MiaAI-Lab:mainfrom
0xdfi:feat/fleet-energy-endpoint

Conversation

@0xdfi

@0xdfi 0xdfi commented Aug 24, 2026

Copy link
Copy Markdown

Summary

sparkDash can now report estimated whole-fleet power, rolling 24-hour and 31-day energy, coverage, a 24-hour hourly-watts graph, and watt-hours per output token through GET /api/fleet-energy.

This PR is stacked on #59. The monitor/freshness commits remain visible here until that prerequisite merges; the fleet-energy capability itself is the final commit.

Related: #59, #64

Design decisions

  • Samples run independently of WebSocket clients every two seconds.
  • A node contributes only when its GPU and CPU collectors explicitly succeeded and both timestamps are at most 10 seconds old.
  • Whole-node watts are estimated from GPU board draw, a bounded CPU utilization model, and fixed platform overhead. Values are marked estimated because this is not a wall meter.
  • Trapezoidal integration stops across gaps longer than 10 seconds. Coverage fields expose missing fleet time instead of silently filling it.
  • Minute buckets persist atomically at mode 0600. The loader rejects corrupt or physically impossible state and prevents restart double-counting.
  • Wh/output-token uses one canonical head and rebases across counter resets, source changes, or observation gaps.

New concepts

Coverage-aware energy integration

Energy is accumulated only between consecutive valid power samples. Each interval also adds coverage time, so clients can distinguish a low-energy period from an unobserved period. This avoids the common but misleading alternative of carrying the last watt value through outages.

Use this approach for telemetry-derived totals when source gaps are expected. Do not use it when a hardware meter already supplies cumulative energy directly.

Validation

  • node --test server/sparks/__tests__/monitor-lifecycle.test.js server/sparks/__tests__/fleet-energy.test.js - 61 passed
  • npm run typecheck - passed
  • npm run build - passed
  • The deployed v1.8 integration preserved the legacy rolling state, reported all four fresh nodes, and served authenticated LCD reads without container restarts or API errors.
  • Code review receipt: complete, no remaining actionable findings.

The unrelated exact-rate timing flakes in the EXL3 and llama.cpp tests are fixed separately in #64.

Post-Deploy Monitoring & Validation

  • Search logs for FleetEnergyTracker, fleet energy sample error, and fleet energy shutdown error for the first hour.
  • Watch freshNodeCount, coverage24hMs, state-file growth, and process event-loop responsiveness.
  • Healthy signal: all configured nodes become fresh, current watts is non-null, coverage advances only while all nodes are valid, and the state file stays mode 0600 across a restart.
  • Failure signal: impossible watts, advancing coverage during a node outage, state-load warnings, persistence errors, or non-zero shutdown caused by failed persistence. Stop the sampler and roll back this PR if observed; retain the state file for diagnosis.
  • Validation window and owner: first hour plus one restart; sparkDash maintainer/operator.

Compound Engineering

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant