[the-gigi-pplx]

Summary

EKS-managed-AMI node groups now stay on their launch-template version when the template changes. Before this change, any edit that produced a new template version rolled them in place. That covers the eks-cluster module's managed-AMI pools (general, cred-authority, and builders when no builders AMI is set or resolved, in modules/space; general in the reference cluster) plus modules/space's general-2 and per-AZ general groups. This adds launch_template[0].version to lifecycle.ignore_changes on aws_eks_node_group.managed, aws_eks_node_group.general_blue_green and aws_eks_node_group.general_per_az.

Every change to what a template launches with bumps its version, including a tag change in common_tags, which the templates copy into their tag specifications. EKS's managed rolling update fails a node whose pods haven't left within about 15 minutes, with PodEvictionFailure. So a general pool running pods with a long termination grace period can't roll in place, and a cluster-wide tag change would fail the roll on every such pool. The rollout-mng module and the legacy rollout-managed worker group (aws_eks_node_group.rollout_managed in modules/space/node_groups.tf) already hold their templates, because an in-place roll of their workers takes hours to days.

ignore_changes can't be conditional, so this applies to every caller of these modules. A group still starts on the template's latest version. After that it moves only deliberately, through either path the new comments document:

• Move the workload to a new group, which starts on the latest version. For general that's the existing blue/green switch to general-2 or the per-AZ groups. A parked pool that has fallen behind is emptied, checked to run no instances immediately before the apply, and then recreated with terraform apply -replace, which brings it back on the latest version, before it starts serving.
• Roll in place with aws eks update-nodegroup-version --launch-template id=<id>,version=<version>, for a group whose pods all leave within EKS's eviction window. --force isn't a way past PodEvictionFailure for the others: it deletes their pods regardless of disruption budgets. Passing the group's current --kubernetes-version and --release-version keeps the call from also upgrading its AMI and Kubernetes version.

The next refresh records the version the group runs, so neither path leaves drift. Until a group is moved, a check block in each module warns about the groups that run an older template version than the latest: managed_node_groups_on_latest_launch_template in eks-cluster and general_node_groups_on_latest_launch_template in space. It's a warning, so it doesn't fail the plan. The plan that changes a template can't name the groups yet, since the new version isn't known until apply; the apply and every later plan do. The template ID isn't held: EKS can't move a group to another template, so a new ID still plans the group's replacement rather than leaving it on a deleted template.

Custom-AMI groups (aws_eks_node_group.custom, which holds builders whenever a builders AMI is set) keep tracking the latest version, so an AMI change still rolls them. worker_capacity is unchanged. The cluster-autoscaler balancing comment now says general groups can stay on different template versions.

reference/README.md now says the reference cluster's general pool doesn't follow its launch template, and gives the command that moves it.

Validation

• terraform fmt -check -recursive infra
• terraform validate on eks/modules/eks-cluster, eks/modules/space, eks/modules/controllers, eks/modules/eks-cluster/examples/minimal and reference/cluster
• tflint on eks/modules/eks-cluster, eks/modules/space, eks/modules/controllers and reference/cluster
• A disposable terraform test against a copy of eks-cluster with mocked providers (not added to the repository). A new group starts on the template's latest version. A template change, modeled by recreating the template under the same ID with a higher latest_version, leaves the group on its old version and raises the check's warning. Replacing the group brings it to the latest version, with no warning. With the hold removed, the second case fails because the group moves to the new version.

No plan against a live cluster was run here. The change adds an ignored attribute, two check blocks and no resources, so an existing group's plan loses any pending launch-template roll and gains a warning while the group is behind.

Severity and risk

Severity: medium. Without it, any change that bumps the shared tags rolls every general pool in place, and on a pool with long-draining pods that roll fails partway.

Risk: a held group keeps launching nodes from its old template version until someone rolls it. It also stops picking up new AMIs as a side effect: the provider's template roll sent no Kubernetes or AMI version, so it could also move these groups to the cluster's Kubernetes version and its newest AMI. Now only update-nodegroup-version does that. A template change that a group needs, such as a new security group, metadata option or disk size, doesn't reach it on apply. That includes a new credential_service_client_security_group_id for cred-authority, whose description now says so. A modules/space caller that runs builders on the EKS-managed AMI (no builders AMI set or resolved) puts builders in the held resource too. Rollback: revert this change. The next apply then rolls every held group whose recorded version is behind the template, in parallel, which can also move it to the cluster's Kubernetes version and its newest AMI. On a pool with slow-draining pods that roll fails the same way this change avoids, so move those pools onto the latest version first.

AI assistance: Claude Code drafted the change and this description.

[cla-assistant]

All committers have signed the CLA.

[nikhilgupta-pplx]

LGTM at f3e6ca9.

• launch_template[0].version hold on managed, general_blue_green and general_per_az matches the existing rollout_managed/rollout-mng pattern. The template ID stays tracked, so a new template still forces replacement instead of pointing at a deleted template.
• Custom-AMI groups (aws_eks_node_group.custom) still track latest, so builder AMI rolls are unchanged.
• Both check blocks resolve: managed iterates only groups that exist (template-only pools are excluded), launch_templates["general"] always exists in modules/space, general-2 can't collide with the general-<zone> keys, and tostring(latest_version) matches the string version the provider refreshes after update-nodegroup-version.
• Dropping the "keep min/max out of an apply that also rolls the group" note is correct. With version/release_version unset and the template version held, Terraform no longer rolls these groups in place.

Before rollout, not blocking: the first apply drops any template roll currently pending on these groups. A group that needs a recent SG or metadata change, e.g. a changed credential_service_client_security_group_id on cred-authority, has to be moved explicitly. The rollback note covers the reverse case: reverting rolls every held group in parallel.

CI is green.