Skip to content

fix(enphase): settle the power buckets and derive load from the balance - #4430

Merged
springfall2008 merged 3 commits into
mainfrom
enphase-settled-power
Aug 3, 2026
Merged

fix(enphase): settle the power buckets and derive load from the balance#4430
springfall2008 merged 3 commits into
mainfrom
enphase-settled-power

Conversation

@springfall2008

Copy link
Copy Markdown
Owner

Fixes two sensor bugs spotted in a deployed system: a saw-toothing pv_power, and a load_power that tracked PV instead of house usage. Independent of #4429.

1. Saw-tooth on the power sensors

interval_power read the 15-minute bucket that had just closed, assuming it was complete. It isn't — the cloud keeps back-filling a bucket for several minutes after it ends, so the first read returns roughly a third of the eventual value and later corrects upward:

09:56  idx=38  674 Wh → 2696 W
10:01  idx=39  243 Wh →  972 W   ← new bucket, still filling
10:07  idx=39  759 Wh → 3036 W   ← same bucket, settled (3.1x)
10:18  idx=40  276 Wh → 1104 W   ← next bucket, drops again

In a 13-hour log, 23 buckets were re-read with a changed value — every one an increase. This affected pv_power, grid_power and battery_power equally, since all three come from the same call; PV was just the most visible.

Fixed by stepping back to a bucket that has stopped changing. Observed revisions only ever touched the just-closed bucket, so one extra step suffices. Cost is up to one extra bucket of lag.

2. load_power was PV

It was published from /app-api/<site>/get_latest_power, which returns production, not consumption. The night-time data is conclusive — it reads 0-1 W from midnight to 04:00, and tracks the PV ramp by day:

Time get_latest_power PV bucket Actual house load
08:51 1759 W 1736 W 232 W
09:29 2363 W 2092 W 252 W
10:07 3105 W 3036 W 260 W

automatic_config wires this to Predbat's load_power arg, which gap-fills load_today — so PV-shaped data was being fed into house-load history.

Load is now the energy-balance residual of the other three channels, from the same settled bucket, so the four sensors agree and a power-flow card balances. That is exactly how the cloud derives its own consumption channel — verified equal to within 1 Wh on 80 of 96 buckets — so reading that channel instead would gain nothing.

Known limitation, deliberately accepted

Being the residual, load absorbs all the timing and rounding skew between the micros, the CT clamps and the battery telemetry. On a site cycling 30 kWh/day through the battery to serve a 5 kWh house load, those terms dwarf the house term and the residual goes unphysical while the battery works hard — around 21% of buckets. It is clamped at zero so it can never render a negative house load, and it is accurate when the battery is calm (1 bad bucket in 8 with the battery idle; a steady 220-260 W through the morning).

Alternatives were measured and rejected:

  • Aggregating — needs a 2-4 hour window before it is reliably plausible, by which point it is no longer house load in any useful sense.
  • Deltaing the daily totals — worse, not better: 99/141 implausible versus 9/43, because the channels restate on independent schedules (a 2995 Wh export increment appeared in one 5-minute poll). Not a precision limit; at ~470 W a 5-minute interval accrues ~39 Wh, so ±1 Wh quantisation is only ±12 W.
  • Enphase's own consumption channel — identical numbers, as above.

load_today is unaffected and remains accurate (buckets sum exactly to the daily total). Documented in docs/components.md, including the 15-30 minute lag now carried by all four power sensors.

The real fix is the Enlighten live MQTT feed (MeterSummaryData over AWS IoT), which carries measured pv/storage/grid/load channels directly. That is being investigated separately; this change is the best available from the /today endpoint in the meantime.

Testing

Five new tests written test-first, covering the settled-bucket selection, the start-of-day clamp, the derived load, the negative-residual clamp, and the consistency invariant (load == pv + grid + battery). One existing test updated — it asserted the old get_latest_power value for load.

./run_pre_commit passes: all hooks green, full quick suite green.

Version bumped v8.47.4 → v8.47.6.

Note: get_latest_power is still fetched and cached but no longer published. It is retained deliberately as the only genuinely live PV figure available — it will be needed to validate the MQTT feed's agg_p_mw scaling.

🤖 Generated with Claude Code

The instantaneous power sensors read the 15-minute energy bucket that had
just closed, on the assumption it was complete. It is not - the cloud keeps
back-filling a bucket for several minutes after it ends, so the first read
returns roughly a third of the eventual figure and later corrects upward.
Every rollover therefore produced a ~3x saw-tooth on pv_power, grid_power
and battery_power alike; in a 13-hour log 23 buckets were re-read with a
changed value, every one an increase. Reading a bucket that has settled
removes it, at the cost of up to one extra bucket of lag.

load_power was published from /app-api/<site>/get_latest_power, which
returns site PRODUCTION, not consumption - it read 0-1 W all night and
tracked the PV ramp by day. That fed PV-shaped data into load_today's
gap filling. It is now the energy-balance residual of the other three
channels, taken from the same settled bucket so the four sensors agree and
a power-flow display balances. This is precisely how the cloud derives its
own consumption channel - verified equal to within 1 Wh on 80 of 96
buckets - so reading that channel instead would gain nothing.

Being the residual, load absorbs the timing and rounding skew between the
micros, the CT clamps and the battery telemetry. On a site cycling 30 kWh
a day through the battery to serve a 5 kWh house load, those terms dwarf
the house term and the residual can go unphysical, so it is clamped at
zero rather than rendering a negative house load. Aggregation does not
rescue it (a 2-4 hour window is needed before it is reliably plausible)
and neither does deltaing the daily totals, which update per channel on
independent schedules and are noisier still. load_today is unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 3, 2026 13:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes two EnphaseCloud monitoring issues in Predbat by (1) deriving instantaneous power from a settled intra-day /today energy bucket to avoid saw-toothing, and (2) publishing load_power as the energy-balance residual of PV/grid/battery so it reflects house usage and keeps the power-flow model consistent.

Changes:

  • Adjust power derivation to step back from the just-closed (still back-filled) bucket and use a settled bucket instead.
  • Replace load_power publishing from get_latest_power with a derived residual (pv + grid + battery) clamped at zero.
  • Update docs and extend Enphase API tests to cover settled-bucket selection, start-of-day clamping, derived load, and negative residual clamping.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
docs/components.md Documents the added 15–30 minute lag for power sensors and the residual-based load_power limitations.
apps/predbat/tests/test_enphase_api.py Adds/updates tests for settled-bucket power selection and derived/clamped load behavior.
apps/predbat/predbat.py Bumps Predbat version to v8.47.6.
apps/predbat/enphase.py Implements settled-bucket selection and derives/publishes load_power from the bucket-based balance.

Comment thread apps/predbat/enphase.py
springfall2008 and others added 2 commits August 3, 2026 15:25
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@springfall2008
springfall2008 merged commit e7b7539 into main Aug 3, 2026
2 checks passed
@springfall2008
springfall2008 deleted the enphase-settled-power branch August 3, 2026 14:10
springfall2008 added a commit that referenced this pull request Aug 3, 2026
The cloud reports a schedule family as scheduleStatus "pending" while a
change settles on the gateway - the normal state straight after any write
Predbat makes. Only "active"/"enabled"/"supported"/"available" counted as
supported, so a site with a perfectly good charge-from-grid family was
judged incapable of it, automatic_config raised and run() returned False:

  Warn: Automatic configuration skipped - Charge-from-grid (CFG)
  scheduling not supported on this site, cannot configure

seen on a site whose cfg family held an active schedule and whose profile
reported scheduleSupported true for both cfg and dtg.

"pending" now counts as supported, and so does any family that actually
holds a schedule, whatever the status string says. "not_supported" still
reports unsupported.

Also aligns the livestream fallback test with the settled-bucket selection
that landed on main in #4430: the helper froze time one bucket too early,
and load now falls back to the energy-balance residual rather than being
left empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
springfall2008 added a commit that referenced this pull request Aug 4, 2026
* feat(enphase): read measured power from the Enlighten livestream

The instantaneous power sensors were derived from the /today 15-minute
energy buckets, which cannot produce a usable house load: consumption is
the residual of much larger terms, so on a site cycling 30 kWh a day
through the battery to serve a 5 kWh house load it is unphysical about a
fifth of the time. load_power was instead published from get_latest_power,
which reports PRODUCTION, not consumption - it reads 0-1 W all night and
tracks the PV ramp by day.

The Enlighten app streams a protobuf DataMsg once a second over
MQTT-on-WebSockets from AWS IoT, carrying separately METERED pv, storage,
grid and load channels plus SOC. Predbat now takes one reading per cycle -
connect, first message, disconnect, the same lifecycle the web app uses -
rather than holding the stream open and re-authorising every 900s.

Credentials are bootstrapped from /pv/aws_sigv4/livestream.json using the
gateway serial, which /today already carries, so no extra discovery call
is needed. AWS IoT's custom authorizer is fed through the MQTT CONNECT
username: the WebSocket takes no query string and no password, because a
browser cannot set custom headers on a WebSocket.

Verified against 379 frames captured from a real session: the channels
satisfy load = pv + grid + battery to 0.0 W on every frame, load reads
154-1979 W where PV reads 4452-4933 W, and the signs already match
Predbat's convention. One of those frames is committed as a test fixture.

The bucket-derived values remain the fallback for pv/grid/battery when the
stream is unavailable, so a failure degrades rather than blanking the
sensors; load is left empty in that case rather than published wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [pre-commit.ci lite] apply automatic fixes

* fix(enphase): keep livestream credentials out of the log

The livestream bootstrap response carries aws_token_value and aws_digest,
the live credentials for the account's AWS IoT stream. Debug API logging
redacted only token/auth_token/access_token, so both were written out in
full - and Predbat logs are routinely shared for debugging. The endpoint
and topic are still logged so the call stays diagnosable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(enphase): treat a pending schedule family as supported

The cloud reports a schedule family as scheduleStatus "pending" while a
change settles on the gateway - the normal state straight after any write
Predbat makes. Only "active"/"enabled"/"supported"/"available" counted as
supported, so a site with a perfectly good charge-from-grid family was
judged incapable of it, automatic_config raised and run() returned False:

  Warn: Automatic configuration skipped - Charge-from-grid (CFG)
  scheduling not supported on this site, cannot configure

seen on a site whose cfg family held an active schedule and whose profile
reported scheduleSupported true for both cfg and dtg.

"pending" now counts as supported, and so does any family that actually
holds a schedule, whatever the status string says. "not_supported" still
reports unsupported.

Also aligns the livestream fallback test with the settled-bucket selection
that landed on main in #4430: the helper froze time one bucket too early,
and load now falls back to the energy-balance residual rather than being
left empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(enphase): publish grid power positive when exporting

Predbat's convention is grid positive on EXPORT - web.py's power flow
reads `grid_power >= 10` as exporting, and sigenergy.py documents the same
- but enphase.py published `import - export`, so every Enphase user's grid
power has been inverted, showing import where the flow diagram expects
export. The livestream's grid channel is negative while exporting too, so
both the measured and the bucket-derived paths needed flipping.

The residual load derivation follows from the sign change: in Predbat's
signs (grid +export, battery +discharge) the balance is pv + battery -
grid, not pv + grid + battery. Load values are unchanged by this; only the
grid sensor's sign moves.

Battery power is left alone: `discharge - charge` (positive on discharge)
already matches the core convention in inverter.py, which detects charging
as `power < -threshold`, and sigenergy's documented mapping. Note that
web.py's power flow reads battery the opposite way (`>= 10` as charging),
which looks like a display bug affecting every integration rather than
something to correct here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(enphase): never republish a stale livestream reading

Livestream readings are instantaneous and carry no usable timestamp of
their own (DataMsg.timestamp is a constant), so keeping one around past
its moment presents an old measurement as current. Two ways that happened:

- Caching. _load_cache restores each key's storage age into data_age, so a
  restart within ENPHASE_REFRESH_POWER left the refresh gate satisfied,
  get_live_power unrun and the restored reading published as live.
  live_power is now in-memory only.
- A failed read left the previous reading in place, so a stream outage
  mid-run republished the last good measurement indefinitely, no restart
  required. A failure now drops it.

Either way the sensors fall back to the bucket values, which lag but are
genuinely current.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(enphase): age livestream readings out instead of dropping them

Clearing the reading on a failed read made a single missed connection flip
all four sensors onto the bucket fallback, which lags 15-30 minutes - a
bigger visible step than simply holding the last measurement a little
longer. A reading now stays in use for ENPHASE_LIVE_MAX_AGE_MINUTES and is
ignored after that, so a blip is absorbed while genuinely old data still
stops being presented as current.

Readings are stamped on arrival because DataMsg.timestamp is a constant
and the payload cannot date itself. They remain in-memory only, so nothing
survives a restart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: pre-commit-ci-lite[bot] <117423508+pre-commit-ci-lite[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants