Skip to content

Add bounded direct ProxySQL capacity probes - #203

Draft
pradeepkintali wants to merge 4 commits into
pkintali/a1-freno-primary-protectionfrom
pkintali/proxysql-capacity-probes
Draft

pradeepkintali wants to merge 4 commits into
pkintali/a1-freno-primary-protectionfrom
pkintali/proxysql-capacity-probes

Conversation

@pradeepkintali

@pradeepkintali pradeepkintali commented Oct 15, 2026 •

Copy link
Copy Markdown

Overview

Add direct ProxySQL frontend and writer-backend capacity probes without extending the existing replica-lag/primary-concurrency rollout PR.

Stacked on #196. The base is pkintali/a1-freno-primary-protection so the diff contains only this capacity feature. It relies on that PR's composite dependencies, fail-closed checks, and recovery behavior. Retarget to the default branch after #196 merges. Nothing is merged or deployed by this PR.

Behavior and load bounds

  • Read the ProxySQL stats interface using an explicit stats-only account; never follow discovered backend addresses to run capacity SQL on MySQL.
  • Frontend: connected clients / explicitly verified frontend connection limit.
  • Writer backend: worst ConnUsed / explicitly verified per-backend limit in the configured writer hostgroup, normally 30. Unknown ONLINE, missing, duplicate, or non-ONLINE expected backends fail the writer probe.
  • Share one reusable admin connection and one five-second success/error snapshot between both metrics for the same endpoint, credentials, and hostgroup. A successful cycle executes two small SELECTs (0.4 queries/second in steady state).
  • Bound the whole batch to one second, limit the pool to one connection, rate-limit failed polling, and retire idle pools.
  • Do not inherit MySQL credentials, metric SQL, HTTP checks, ignored-host behavior, or fail-open no-host behavior. Reject common application ports and clear capacity rosters when discovery fails.
  • Support explicitly selected colocated HAProxy/Vitess primary discovery or an explicit static proxy roster; override discovered database ports with the admin port.

Safety / limitations

No listeners, firewall rules, production credentials, runtime limits, or enforcement are changed here. Paired configuration PR: https://github.com/github/puppet/pull/44385 (separate, disabled-by-default rollout).

Stats-only credentials cannot read runtime configuration tables. Limits must be verified against runtime for every endpoint and kept synchronized when membership/configuration changes; there is no runtime-limit drift detection. Example limits/thresholds are not production recommendations.

No backend SQL or connections are added by these probes, but colocated ProxySQL still shares CPU/memory with MySQL. Production CPU/latency overhead has not been benchmarked and must be reviewed in shadow mode before enforcement.

Validation

  • Full repository script/cibuild passed in the Go 1.20 container: binary build and race-enabled tests for all packages.
  • Focused race-enabled config, sampler, MySQL-dispatch, aggregation, and discovery regressions passed.
  • Real ProxySQL 2.7.3 stats-only integration passed in an isolated network: frontend/backend reads succeeded and the listening backend sentinel observed zero connections. No ports were published; the fixture was stopped and removed.
  • Both simulated Puppet shadow and selective-enforcement production renders were accepted by the actual Freno configuration loader, with strict admin-only settings.

Rollout / rollback

Deploy the binary with existing configuration first. Collection remains absent unless ProxySQLCapacitySettings is explicitly configured. Approve admin access/credential transport, verify limits/endpoint identity, observe query load, validate recovery/client defer behavior, then canary one cluster's required dependencies.

Remove both capacity dependencies to roll back enforcement, retaining lag and primary concurrency. Remove the separately configured probes to stop capacity collection; coordinate configuration restarts and startup recovery.

Pradeep Kintali and others added 4 commits October 15, 2026 11:01
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: dea961fa-5766-415f-a0f5-2562c55a0385
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d
Carry shadow graph evaluation, isolated HTTP and background counters, startup notices, and public rollout guidance into the capacity stack while preserving direct capacity sampling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fbc83df2-d1af-406f-b97b-51cf0aaa570d

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant