🐛 (ZENKO-5303) make voting and priority right

[DarkIsDude]

#2449

UPSTREAM : bitnami/containers#95156

What does this PR do, and why do we need it?

It fixes a bug in the MongoDB sharded image bootstrap (libmongodb.sh) that can leave every replica set with only one voting member, turning that single node into a hard single point of failure: if it goes down, the shard (and therefore the datastore) goes read‑only and Zenko is down.

---

A 30‑second MongoDB primer (for non‑Mongo readers)

Our data lives in replica sets: groups of MongoDB nodes (here, 3 per shard and 3 for the config server) that each hold a copy of the data.

• One node is the PRIMARY (takes writes); the others are SECONDARIES (copies).
• To stay alive, a replica set must keep a majority of votes online and be able to elect a PRIMARY. Two settings per member matter:
  • votes (0 or 1): can this member vote in an election? Majority is counted over the sum of votes.
  • priority (0+): can this member be elected PRIMARY? priority: 0 means "never become PRIMARY".
• A healthy 3‑node set has 3 members with votes: 1, priority: 1. It survives losing any one node: the remaining 2 votes are still a majority, and a surviving member can be elected PRIMARY.

A member that is only partially configured — votes: 1, priority: 0, or worse votes: 0, priority: 0 — still holds data but cannot help keep the set alive.

---

The bug

When a SECONDARY first joins, the bootstrap script does this (mongodb_configure_secondary):

1. rs.add(... votes: 0, priority: 0) — add the node without voting power so it doesn't disturb the existing majority while it copies data (this is MongoDB's recommended safe procedure).
2. Confirm the node is now listed in the replica set (mongodb_node_currently_in_cluster).
3. Wait for it to finish its initial sync (reach SECONDARY state).
4. Grant voting rights: reconfigure it to votes: 1, priority: 1.

The confirmation in step 2 reads rs.status() and greps the output for the node:

result=$(mongodb_execute ... <<< "rs.status().members")
grep -q "'$node:$port'" <<<"$result"

The problem: with the broken image, mongodb_execute is a thin wrapper around debug_execute, which throws away stdout unless BITNAMI_DEBUG=true (it isn't, by default):

debug_execute() { if is_boolean_yes "${BITNAMI_DEBUG:-false}"; then "$@"; else "$@" >/dev/null 2>&1; fi; }

So result is always empty, the grep always fails, and mongodb_node_currently_in_cluster always returns false. That makes step 2 (mongodb_wait_confirmation) time out and the bootstrap aborts with:

ERROR ==> Unable to confirm that <node> has been added to the replica set!

The container exits, Kubernetes restarts the pod, and on the second boot the data directory already exists, so the bootstrap takes the "deploy with persisted data" path and skips replica set configuration entirely. The node is left frozen at votes: 0, priority: 0. Steps 3–4 never run, so the SECONDARY is never promoted.

Net result: only the bootstrap PRIMARY (*-0) ends up with a vote → 1 voter per replica set → single point of failure.

---

Root cause: an incomplete fork of the Bitnami image

Bitnami stopped publishing the mongodb-sharded image, so we vendored its scripts into the repo (ZENKO-5110, #2366). The vendoring happened in two commits, and they are not equal:

Commit	Branch	libmongodb.sh	mongodb_execute() defs	Result
0ae8c8a3	development/2.14	1712 lines	2	✅ works (→ 2.14.5)
2f6c7e42	development/2.15	1669 lines	1	❌ broken (→ 2.15.1)

Compare: the only relevant difference is the last 43 lines of libmongodb.sh.

Upstream's libmongodb.sh is assembled by concatenating script fragments, and it deliberately defines mongodb_execute twice:

1. Early in the file — the legacy wrapper that discards stdout (debug_execute mongodb_execute_print_output "$@").
2. As a final appended fragment (it carries its own # Copyright … header and # shellcheck disable=SC2148 — the tell‑tale "no shebang" marker of a separate concatenated file) — the real version that calls mongosh directly and returns output.

In bash the last definition wins, so upstream's effective mongodb_execute is #2 (returns output) — which is exactly what mongodb_node_currently_in_cluster needs.

The 2.15 re‑vendoring (2f6c7e42) truncated the file at 1669 lines and dropped that final fragment. Only the output‑discarding wrapper was left, silently reverting mongodb_execute and breaking the confirmation check. The 2.14 vendoring had copied the whole file, so 2.14.5 worked. We didn't add a fix in 2.14 — we just vendored completely there, and lost it in 2.15.

It was easy to miss because dropping a duplicate function definition leaves valid bash that runs fine; the only signal was the line count (1712 vs 1669), and the failure only surfaces as a silent bootstrap race that is invisible until a node dies.

How we confirmed it (two clusters, same Mongo version)

A cluster on image base 2.14.5 was healthy (3 voters); a cluster on 2.15.1 was broken (1 voter). We confirmed the chain end‑to‑end from the live rs.conf() (broken cluster: secondaries at votes: 0, priority: 0; healthy cluster: votes: 1) and the pod boot logs (broken cluster fails at "Unable to confirm…"; healthy one gets past it). The diff between the two images' libmongodb.sh was exactly the 43‑line fragment above.

---

The fix

Two commits:

1. 🐛 add missing libmongo.sh fork — restores the dropped 43‑line fragment, so the file matches upstream again. mongodb_execute is once more the output‑returning definition, mongodb_node_currently_in_cluster can read rs.status(), and the bootstrap no longer aborts before granting voting rights. This is the root‑cause fix (faithful re‑sync with upstream). (in the first PR)

2. 🐛 ensure secondary keeps votes and priority after restart — makes the voting‑rights grant idempotent and re‑runnable, so a node can no longer be left stranded without votes:

   • new helper mongodb_secondary_node_has_voting_rights checks whether the member already has votes > 0 && priority > 0;
   • mongodb_configure_secondary now grants voting rights whenever they are missing — even if the node is already in the cluster, instead of only on the freshly‑added path. So if a previous attempt added the node at votes/priority 0 and then failed (or was restarted) before promotion, the next run finishes the job and converges it to votes: 1, priority: 1;
   • the misleading error message ("did not get marked as secondary" printed for the voting step) is corrected to "did not get granted voting rights".

After this change a fresh deployment reliably ends with all members at votes: 1, priority: 1 (true HA), and a member that is already in the replica set but under‑privileged is repaired rather than silently left non‑voting.

---

Which issue does this PR fix?

Fixes ZENKO-5302.

Special notes for your reviewers:

• Already‑broken clusters won't fully self‑heal from the image alone. On a pod restart the bootstrap takes the persisted‑data path, which skips replica‑set (re)configuration entirely, so an existing cluster already stranded at votes: 0 still needs a one‑time manual rs.reconfig() on each replica set (shard-N and configsvr) to set members 1 and 2 to votes: 1, priority: 1. The script change guarantees correct behaviour for new bootstraps and for any path where mongodb_configure_secondary runs against a member that lacks voting rights.
• Should we keep the second fix ? We can use a dedicated PR for that as we already have this issue on older cluster (4.2.3 have it).

[francoisferrand]

The change itself looks ok; however I am not sure we should do this.

I think the configuration was done only on first startup by design : this ensures that after first startup, each instance will just use the configuration it has and let (human) operator manage it.

This means there is a gap in the chart indeed: if the instance is restarted during this startup, it may be left in an incorrect state... But this is a tradeoff: either we enforce the state of replicas (i.e. your change) -even though we don't actually know the intent of the human deploying the chart- and risk setting the wrong state for more advanced setups ; or we keep the existing/upstream approach to automatically handle the nominal path only, and leave recovery for humans...

→ if we were writing an operator, I would say it should recover automatically -and we just need to add whatever necessary in the CR to make intent clear
→ however this is a chart -mostly static, with not much way to get instant- so I would rather stay conservative
→ practically we never experienced this issue until we removed part of the chart (what you already fixed) : so the risk seems very low to keep it as upstream, and we can add an extra check in installer to validate the overall state during/after chart deployment for extra safety?

[delthas]

• Makefile clone optimization: LGTM
• Granting voting rights to existing replicas: will break on > 7 replicas (as mentioned by François). Not sure how to proceed

[francoisferrand]

Granting voting rights to existing replicas will break on > 7 replicas : need to add extra condition to either

• skip mongodb_configure_secondary_node_voting if there are already too many voters
• or detect (and ignore) the "too many voters" error
• or ignore the mongodb_configure_secondary_node_voting error if we see (after the call/error) that there are 7 voters already

(in particular, must check the behavior with the 9-replicas setup we have: to ensure the change would not change the voters: which are manually set to be precisely in the expected datacenter/room, and thus meet the availability constraints specified by customer)

[SylvainSenechal]

. nvm

[DarkIsDude]

Granting voting rights to existing replicas will break on > 7 replicas : need to add extra condition to either

(in particular, must check the behavior with the 9-replicas setup we have: to ensure the change would not change the voters: which are manually set to be precisely in the expected datacenter/room, and thus meet the availability constraints specified by customer)

Tested on a real 9-replica shard, on both 2.15 and this branch.

Good catch, but the check was worse than suspected: it grepped mongosh output for true, which always matches the directConnection=true banner — so no secondary ever got votes at all. Fixed in f313b49, plus 5f71df4 for the same bug in mongodb_is_secondary_node_ready.

The >7 crash is real but pre-existing (reproduced on 2.15): ~2 min of retries, exit 1, then the restart finds a non-empty data dir, skips config, and the member settles as non-voting — final topology is still 9 members / 7 voters.

Voters can't drift in the nominal case: configure_replica_set only runs when the data dir is empty (libmongodb-sharded.sh:74), so restarts and rolling updates never touch rs.conf(); and at 7 voters the ceiling rejects any promotion.

The only real risk is "empty disk + ≤6 voters" (6+1=7 is accepted), i.e. node replacement, since storage is node-local. We can add your idea — a pre-check on the voter count rather than parsing the error string if you prefer but this will differ from the upstream ?

[francoisferrand]

The history of this PR seems weird : there are commits to remove "vendored image dirs ahead of subtree bootstrap"

The whole point of using git subtree is that we don't revenger from scratch (= loose local changes), but can instead repeatedly merge with the usual git semantics

→ please remove the history, and update upstream with merge only (bootstrap has been done already, no need to redo it

[DarkIsDude]

The history of this PR seems weird : there are commits to remove "vendored image dirs ahead of subtree bootstrap"

The whole point of using git subtree is that we don't revenger from scratch (= loose local changes), but can instead repeatedly merge with the usual git semantics

→ please remove the history, and update upstream with merge only (bootstrap has been done already, no need to redo it

Thanks for the question. The bootstrap was never done or working. The subtree was never bootstrapped for solution-base/images/* but for solution-base/mongodb/charts/mongodb-sharded. The initial copy was just a plain copy 😬. I did a reword in any case to make it more clear.

[delthas]

Two threads on solution-base/images/Makefile covering the bootstrap baselines and the vendor-sync breakage.

[delthas]

This is the root cause of the subtree trouble, and it's why the charts prefix needs none of this ceremony.

solution-base/mongodb/Makefile:35 fetches full history deliberately:

# Fetch full main + the target chart tag so `git subtree split` can synthesize
# the chart's history.
git fetch --no-tags $(BITNAMI_REMOTE) main

With full history, git subtree split is deterministic and chained — re-splitting a newer upstream ref regenerates the previous split commit as an ancestor of the new one, so the git-subtree-split: hash recorded in history always resolves locally. That's why e0089dd84c is a real incremental sync (Squashed '…' changes from d2eb70018d..cd87843fdd) with nothing pushed anywhere.

--depth 1 breaks it: each shallow split is an unrelated single root, so the recorded hash can never be regenerated. subtree merge --squash re-resolves it through process_subtree_split_trailer, and since cmd_merge runs without a repository argument it cannot even attempt a fetch — the split exists only on a local vendor/* branch, and git ls-remote --heads origin 'refs/heads/vendor/*' is empty. Reproduced on a scratch repo:

$ git subtree merge --prefix=images/img vendor2 --squash
fatal: could not rev-parse split hash fc8ac3d4… from commit f06304e9…

So make vendor-sync fails on any other clone, and here too as soon as a pin moves — immediate for the two main refs above.

Fix is one line: drop --depth 1, keep --filter=blob:none. A blobless full-history fetch still regenerates the identical split commit and the merge resolves, so most of the speed win survives and this Makefile ends up matching the charts one. The ~4-minute split cost is already documented in how_to_upgrade.md.

[francoisferrand]

👍 for the makefile bug, but not clear to me where to go from there ( too long/complicated explications, and I'm lost in ordering of multiple threads...)

• with this fix, can we directly perform the update from bitnami upstream, or do we still need the "re-bootstrap" commits ? (= do we still need the "rebootstrap / merge / restore changes" commits)
• for the SHA1 : it should actually be embedded in git history (i.e. the git (subtree) merge), and the only moment where we run the makefile target is to merge upstream...i.e. pick up newest changes. My understanding is thus that these command are not about being reproducible (except when upstream did not change), but really about doing the upgrade / upstream sync. Is this it, or am I missing something?

[DarkIsDude]

Fixed, but not by dropping --depth 1 — I couldn't reproduce it as an ancestry problem. Two independent --depth 1 splits give unrelated roots and merge --squash still works: it resolves the recorded trailer as an object and diffs two trees, never consulting ancestry.
The real cause is reachability — the vendor/* branches were local-only, which is the empty ls-remote you saw. So update-vendor-branch-% now pushes each split to refs/heads/vendor-baseline// and vendor-sync-% fetches them before merging. All three baselines are published and match the trailers in history.
Keeping --depth 1 for cost: subtree split doesn't restrict its rev-list to the prefix, so full history walks all 485k commits of the containers monorepo. Happy to switch if you think unrelated roots are a problem in themselves.

Yes, it's the goal of the fix to be able to run it without redoing the bootstrap (the goal is to do what was the first goal that was not done finally).

For the SHA1 the goal of the makefile is mainly to update, but indeed not reproductible. I changed the makefile to make it reproductible 🙏

[francoisferrand]

• I think @benzekrimaha did the initial bootstrap : if she still has the branch (locally), she could just push it so we don't need the bootstrap?
• I changed the makefile to make it reproductible : I don't understand the benefit, especially when it contradicts the goal of the makefile (i.e. to update the charts). Can you please elaborate?

[delthas]

Sorry I had to defer to Claude largely here for the comments because the subtree stuff is hard to explain easily

[francoisferrand]

there is no rush there, should it not go in 2.16 instead?

[francoisferrand]

• I think @benzekrimaha did the initial bootstrap : if she still has the branch (locally), she could just push it so we don't need the bootstrap?
• I changed the makefile to make it reproductible : I don't understand the benefit, especially when it contradicts the goal of the makefile (i.e. to update the charts). Can you please elaborate?

[francoisferrand]

since you now stick the SHA1 ("changed the makefile to make it reproductible"), what is now the process to bump ? (i.e. fetch upstream, update the SHA1 in makefile, ...)
→ shoud update the docs/readme

[delthas]

I'd say update the refs, then run make vendor-sync, similarly to charts where we bump CHART_VERSION and run make?