K8SPG-1057: Allow using etcd as patroni DCS

[yoav-katz]

CHANGE DESCRIPTION

Problem:
Patroni supports multiple DCS backends, but the operator hardcodes Kubernetes Endpoints as the only option. This blocks clusters on managed Kubernetes platforms where workloads cannot reach the control plane API.

Cause:
The kubernetes: stanza was hardcoded in the generated Patroni config with no mechanism to select a different backend.
Several other pieces of the operator also assumed k8s DCS: RBAC rules unconditionally granted Endpoints permissions, the primary service routed through Patroni-managed Endpoints objects, and pod role labels/annotations were expected to be set by Patroni itself (which only happens with k8s DCS).

Solution:
Add a spec.patroni.dcs field (type: kubernetes default, type: etcd alternative). The field is immutable after cluster creation, enforced by a CEL validation rule on the CRD.

When type: etcd, the operator:

• Emits an etcd3: stanza in the generated Patroni config instead of kubernetes:, with optional TLS (cacert/cert/key) and auth credentials (PATRONI_ETCD3_USERNAME/PATRONI_ETCD3_PASSWORD) sourced from referenced Secrets.
• Injects on_start and on_role_change Patroni callbacks pointing to a new patroni-role-change.sh script. Since Patroni does not set pod role labels or the status annotation when using etcd DCS, this script patches the pod via the k8s API on every role transition, restoring the label (role=primary|replica) and annotation ({"role":"primary"}) that the rest of the operator depends on for Service routing and primary detection.
• Creates the primary Service with a label selector (role=primary) instead of the previous headless-Endpoints-to-Patroni-leader-ClusterIP indirection, which only works with k8s DCS.
• Skips creating the Patroni leader lease Service and distributed configuration Service, which are k8s DCS artifacts.
• Omits the Endpoints RBAC permissions from the postgres pod ServiceAccount, since they are not needed.
• Validates that referenced TLS and auth Secrets exist and contain the required keys, surfacing issues as Warning events on the cluster.

The Kubernetes DCS path is unchanged.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Helm Chart Merge Request
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version?
• Does the change support oldest and newest supported Kubernetes version?

[it-percona-cla]

All committers have signed the CLA.

[yoav-katz]

I will wait for the jira ticket to open the PR in the helm charts repo

[yoav-katz]

and another note - this is my first OSS contribute so be gentel 😄
if there is stuff that you think should be changed becuase of style/dependency consideration I will be happy to fix!

[egegunes]

@yoav-katz the implementation looks good to me in general. but we definitely need an e2e test that deploys etcd and configures PerconaPGCluster to use it.

[DanBrima]

would love to see this merged!

[Copilot]

Pull request overview

This PR adds support for using external etcd as Patroni’s DCS backend (in addition to the existing Kubernetes Endpoints DCS), enabling deployments where workloads cannot reach the Kubernetes control plane API. It introduces a new spec.patroni.dcs API, updates Patroni config generation, RBAC/service behavior, and adds validations plus unit/E2E coverage for the etcd path.

Changes:

• Add spec.patroni.dcs (default kubernetes, optional etcd) with CEL immutability validation and generated deepcopy/CRD updates.
• When etcd DCS is selected: generate etcd3: Patroni config, add role-change callbacks, adjust primary Service selection logic, and validate referenced TLS/auth Secrets (with Secret watch/indexing).
• Add unit tests and a new KUTTL E2E scenario (etcd-dcs) to validate config, behavior, and immutability.

Reviewed changes

Copilot reviewed 41 out of 42 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pkg/apis/upstream.pgv2.percona.com/v1beta1/zz_generated.deepcopy.go	Generated deepcopy updates for new Patroni DCS types.
pkg/apis/upstream.pgv2.percona.com/v1beta1/patroni_types.go	Adds dcs API types, helpers, and CEL validations.
pkg/apis/pgv2.percona.com/v2/perconapgcluster_types.go	Adds etcd DCS validation and Secret indexer for reconciliation triggers.
pkg/apis/pgv2.percona.com/v2/perconapgcluster_types_test.go	Unit tests for Percona CR validation of etcd DCS endpoints.
percona/controller/pgcluster/patroni_etcd.go	Reconcile-time Secret presence/key validation + Warning events for etcd DCS Secrets.
percona/controller/pgcluster/patroni_etcd_test.go	Unit tests for etcd DCS Secret validation reconciliation behavior.
percona/controller/pgcluster/controller.go	Watches Secrets via multiple field indexes (envFrom + patroni etcd secrets) with dedupe.
internal/patroni/reconcile.go	Mount etcd TLS Secret and inject Patroni etcd auth env vars into instance Pods.
internal/patroni/rbac.go	Conditionalize Endpoints/Service create RBAC rules based on DCS type.
internal/patroni/config.go	Emit etcd3: config + callbacks for etcd DCS; keep Kubernetes DCS behavior unchanged.
internal/patroni/config_test.go	Adds unit coverage for etcd DCS Patroni YAML generation and env behavior.
internal/controller/postgrescluster/patroni.go	Skip k8s-DCS-specific artifacts/status reads when using etcd DCS.
internal/controller/postgrescluster/cluster.go	Use selector-based primary Service for etcd DCS; avoid applying Endpoints when not used.
e2e-tests/tests/etcd-dcs/00-deploy-operator.yaml	E2E setup step: deploy operator and client.
e2e-tests/tests/etcd-dcs/00-assert.yaml	E2E assertions for operator/CRD readiness.
e2e-tests/tests/etcd-dcs/01-etcd-setup.yaml	E2E step: deploy a single-node etcd StatefulSet for testing.
e2e-tests/tests/etcd-dcs/01-assert.yaml	E2E assertion: etcd is ready.
e2e-tests/tests/etcd-dcs/02-create-cluster.yaml	E2E step: create cluster configured for etcd DCS.
e2e-tests/tests/etcd-dcs/02-assert.yaml	E2E assertions: cluster reaches ready state with etcd DCS.
e2e-tests/tests/etcd-dcs/03-write-data.yaml	E2E: write data to primary via client.
e2e-tests/tests/etcd-dcs/04-read-from-primary.yaml	E2E: read data back from primary.
e2e-tests/tests/etcd-dcs/04-assert.yaml	E2E assertion: expected read result.
e2e-tests/tests/etcd-dcs/05-assert.yaml	E2E assertions for created resources.
e2e-tests/tests/etcd-dcs/06-check-patroni-config.yaml	E2E: verify Patroni config contains etcd3 + callbacks and omits kubernetes:.
e2e-tests/tests/etcd-dcs/07-check-patronictl.yaml	E2E: verify patronictl list shows running/leader.
e2e-tests/tests/etcd-dcs/08-check-etcd-keys.yaml	E2E: verify Patroni keys appear in etcd.
e2e-tests/tests/etcd-dcs/09-check-no-warning-events.yaml	E2E: ensure no unexpected Warning events for etcd secrets.
e2e-tests/tests/etcd-dcs/10-check-dcs-immutability.yaml	E2E: ensure DCS type immutability is enforced by admission/CEL.
e2e-tests/tests/etcd-dcs/11-check-pod-labels.yaml	E2E: verify role labels are present (callback executed).
e2e-tests/tests/etcd-dcs/99-remove-cluster-gracefully.yaml	E2E teardown: remove resources and validate operator stability.
e2e-tests/run-release.csv	Adds etcd-dcs to release E2E run list.
e2e-tests/run-pr.csv	Adds etcd-dcs to PR E2E run list.
deploy/cw-bundle.yaml	Bundle CRD updates to include new DCS schema/validations.
deploy/crd.yaml	Generated CRD updates for new DCS schema/validations.
deploy/bundle.yaml	Bundle CRD updates for new DCS schema/validations.
config/crd/bases/upstream.pgv2.percona.com_postgresclusters.yaml	Base CRD schema updates for DCS fields/validations.
config/crd/bases/pgv2.percona.com_perconapgclusters.yaml	Base CRD schema updates for DCS fields/validations.
cmd/postgres-operator/main.go	Adds field index registration for etcd DCS referenced Secrets.
build/postgres-operator/patroni-role-change.sh	New Patroni callback script to patch pod role label + status annotation.
build/postgres-operator/init-entrypoint.sh	Installs the new Patroni role-change script into runtime bindir.
build/postgres-operator/Dockerfile	Ships the new Patroni role-change script in the image.
build/crd/percona/generated/pgv2.percona.com_perconapgclusters.yaml	Generated Percona CRD output updated for new DCS schema/validations.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 42 changed files in this pull request and generated 2 comments.

[yoav-katz]

@egegunes Im not sure what caused this failure, my commit only touched unit test files so lint will pass😅
Should I open an issue?
Is there a way to retrigger only this specific test?

[egegunes]

It seems to fail on cluster deletion due to timeout. I doubt it's something caused by these changes. I retriggered it.

[yoav-katz]

@mayankshah1607
Im adding a One-shot Kubernetes Job (same pattern as the existing pgBackRest restore Job) using the Patroni image + its config/secrets, whose only command is patronictl remove <scope> (confirmation piped via stdin). observeRestoreEnv looks it up like it does restoreJob; prepareForRestore creates it once pods are gone and waits for it to complete before resetting status.

[JNKPercona]

Test Name	Result	Time
backup-enable-disable	passed	00:19:32
builtin-extensions	passed	00:08:04
cert-manager-tls	passed	00:12:47
custom-envs	passed	00:23:44
custom-tls	passed	00:07:34
database-init-sql	passed	00:02:42
demand-backup	passed	00:25:08
demand-backup-offline-snapshot	passed	00:13:27
dynamic-configuration	passed	00:03:43
finalizers	passed	00:05:54
init-deploy	passed	00:03:27
huge-pages	passed	00:03:06
major-upgrade-14-to-15	passed	00:14:36
major-upgrade-15-to-16	passed	00:11:15
major-upgrade-16-to-17	failure	00:15:01
major-upgrade-17-to-18	passed	00:10:55
ldap	passed	00:05:35
ldap-tls	passed	00:09:41
monitoring	passed	00:08:58
monitoring-pmm3	passed	00:09:18
one-pod	passed	00:06:05
operator-self-healing	passed	00:10:36
pitr	passed	00:14:04
scaling	passed	00:06:58
scheduled-backup	passed	00:28:28
self-healing	passed	00:09:46
sidecars	passed	00:03:04
standby-pgbackrest	passed	00:20:35
standby-streaming	passed	00:13:15
start-from-backup	passed	00:11:33
tablespaces	failure	00:17:52
telemetry-transfer	passed	00:04:28
upgrade-consistency	passed	00:11:11
upgrade-minor	passed	00:07:38
users	passed	00:05:14
etcd-dcs	passed	00:04:05
migration-from-crunchy-standby	passed	00:18:52
migration-from-crunchy-pv	passed	00:08:56
migration-from-crunchy-backup-restore	passed	00:18:32

Summary	Value
Tests Run	39/39
Job Duration	02:49:35
Total Test Time	07:15:59

commit: 2f836f1
image: perconalab/percona-postgresql-operator:PR-1647-2f836f1ab