Skip to content

USHIFT-7019: C2CC - Increase amount of interconnected VMs in from 2 to 3#6706

Open
vanhalenar wants to merge 8 commits into
openshift:mainfrom
vanhalenar:2-to-3
Open

USHIFT-7019: C2CC - Increase amount of interconnected VMs in from 2 to 3#6706
vanhalenar wants to merge 8 commits into
openshift:mainfrom
vanhalenar:2-to-3

Conversation

@vanhalenar
Copy link
Copy Markdown
Contributor

@vanhalenar vanhalenar commented May 20, 2026

Increase amount of interconnected VMs in from 2 to 3 in C2CC RF tests.

Summary by CodeRabbit

  • Tests

    • Added Cluster C to all c2cc test suites and kubeconfig outputs.
    • Replaced many per-cluster cases with template-driven tests validating cross-cluster pod/service connectivity, sanity, and infrastructure across A/B/C.
    • Expanded cleanup and validation checks to include Cluster C CIDRs and extended readiness/connectivity waits to cover Cluster C.
  • Chores

    • Updated test resources and provisioning/orchestration to prepare, configure, include, and remove the additional cluster and its network/kubeconfig settings.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 20, 2026
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented May 20, 2026

@vanhalenar: This pull request references USHIFT-7019 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Increase amount of interconnected VMs in from 2 to 3 in C2CC RF tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 20, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds a third remote cluster (cluster-c): new Robot variables, scenario provisioning and VM lifecycle for host3, refactored c2cc config to support multiple remotes, kubeconfig collection, and Robot suites refactored to run template-driven tests across clusters A/B/C.

Changes

Cluster C Test Infrastructure and Suite Updates

Layer / File(s) Summary
Test variables and runner propagation
test/resources/c2cc.resource, test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh, test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
Declare Cluster C variables (${CLUSTER_C_POD_CIDR}, ${CLUSTER_C_SVC_CIDR}, ${CLUSTER_C_DOMAIN}, ${HOST3_*}, ${KUBECONFIG_C}) and pass them into the test-runner invocation.
C2CC configuration refactor
test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh, test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
Refactor configure_c2cc_host to accept repeated remote-cluster tuples, build combined clusterToCluster.remoteClusters YAML, apply firewall allow-source rules, and configure each host so A/B/C reference each other.
VM lifecycle and kubeconfig collection
test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh, test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh
Prepare and launch host3, inject clusterNetwork/serviceNetwork for cluster C into host3 post-microshift config, include host3 in removal, wait for MicroShift readiness, and copy kubeconfig-c to scenario outputs.
Connectivity suite refactor
test/suites/c2cc/connectivity.robot
Replace explicit pairwise A↔B tests with Test Cross Cluster Connectivity template; add Test Connectivity Between Clusters keyword and extend setup/deploy/wait/cleanup loops to include cluster-c.
Infrastructure suite refactor
test/suites/c2cc/infrastructure.robot
Consolidate per-cluster infra checks into template-driven validations across A/B/C (routes, IP rules, nftables bypass, OVN static routes, SNAT-exclude) and register cluster-c in Setup using HOST3_* and KUBECONFIG_C.
Sanity and Cleanup updates
test/suites/c2cc/sanity.robot, test/suites/c2cc/cleanup.robot
Convert sanity checks to parameterized templates that run for cluster-a/b/c and register cluster-c during suite setup for sanity and cleanup.

Sequence Diagram (high-level)

sequenceDiagram
  participant Runner
  participant ScenarioScript
  participant Host1
  participant Host2
  participant Host3
  participant RobotSuites
  Runner->>ScenarioScript: create & configure hosts (host1, host2, host3)
  ScenarioScript->>Host3: inject cluster C networks and launch MicroShift
  ScenarioScript->>Host1: configure C2CC routes for A/B/C
  ScenarioScript->>Host2: configure C2CC routes for A/B/C
  ScenarioScript->>Host3: configure C2CC routes for A/B/C
  ScenarioScript->>ScenarioScript: collect kubeconfig-a/b/c
  Runner->>RobotSuites: execute template-driven tests across cluster-a/b/c with KUBECONFIG_C and CLUSTER_C_* variables
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers

  • jerpeter1
🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: extending C2CC tests from 2 to 3 interconnected VMs, which is reflected throughout all modified files.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR contains only Robot Framework tests and shell scripts; no Ginkgo Go tests were modified, so the check for stable Ginkgo test names is not applicable.
Test Structure And Quality ✅ Passed PR contains only Robot Framework and shell script tests, not Ginkgo tests. Custom check for Ginkgo test quality is not applicable.
Microshift Test Compatibility ✅ Passed PR contains only Robot Framework tests and shell scripts, no Ginkgo e2e tests. Custom check only applies to Ginkgo tests, so it is not applicable here.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR modifies only Robot Framework tests (.robot), shell scripts (.sh), and resource files (.resource). No Ginkgo e2e tests are added, so SNO compatibility check does not apply.
Topology-Aware Scheduling Compatibility ✅ Passed PR modifies only test infrastructure: Robot Framework test suites, bash scripts, test resources, and simple Pod/Service manifests with no scheduling constraints or affinity rules.
Ote Binary Stdout Contract ✅ Passed PR contains no Go code or OTE binaries; only Robot Framework tests (.robot), resource files (.resource), and shell scripts (.sh) that cannot violate stdout contracts.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR modifies only Robot Framework test suites and shell scripts; no Ginkgo e2e tests are added, making the check inapplicable.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from copejon and pmtk May 20, 2026 13:04
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 20, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: vanhalenar
Once this PR has been reviewed and has the lgtm label, please assign pmtk for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
test/suites/c2cc/cleanup.robot (1)

89-89: ⚡ Quick win

Cleanup assertions should cover Cluster C CIDRs too.

Now that cluster-c is registered, cleanup checks should also assert Cluster C CIDRs/rules are removed from cluster-a; currently only Cluster B is validated.

✅ Minimal assertion additions
 Should Not Contain    ${stdout}    ${CLUSTER_B_POD_CIDR}
 Should Not Contain    ${stdout}    ${CLUSTER_B_SVC_CIDR}
+Should Not Contain    ${stdout}    ${CLUSTER_C_POD_CIDR}
+Should Not Contain    ${stdout}    ${CLUSTER_C_SVC_CIDR}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/suites/c2cc/cleanup.robot` at line 89, After registering cluster-c (the
line using "Register Remote Cluster    cluster-c    ${HOST3_IP}   
${HOST3_SSH_PORT}    ${KUBECONFIG_C}"), add the same cleanup assertions that
currently validate Cluster B's CIDRs/rules are removed from cluster-a but
applied to Cluster C: duplicate the existing Cluster B cleanup checks and change
their targets from cluster-b to cluster-c and from ${KUBECONFIG_B} (if used) to
${KUBECONFIG_C}, ensuring the assertions verify Cluster C CIDRs/rules are
removed from cluster-a.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/scenarios-bootc/el10/presubmits/el102-src`@c2cc.sh:
- Around line 66-71: The current configure_c2cc_host function overwrites
50-c2cc.yaml on each call so per-host remote config loses earlier entries;
modify configure_c2cc_host to collect or render both remote cluster entries for
a given host and write a single 50-c2cc.yaml containing both remote definitions
(instead of writing once per remote), and perform a single restart/apply after
that write; update logic where configure_c2cc_host is invoked for
host1/host2/host3 so each host’s call produces a combined file (or append-safe
rendering) rather than successive overwrites.

In `@test/suites/c2cc/sanity.robot`:
- Around line 17-43: The suite defines tests that reference "cluster-c" (e.g.,
test cases "Cluster Is Running", "All Pods On Cluster Are Ready", "Cluster Has
Expected Node", "C2CC Controller Is Running On Cluster") but setup never
registers that cluster, causing "Oc On Cluster    cluster-c" lookups to fail;
update the suite setup to register/define cluster-c alongside cluster-a and
cluster-b (where clusters are registered for Oc On Cluster/template lookups) so
the templates and the Oc On Cluster keyword can resolve cluster-c during test
execution.

---

Nitpick comments:
In `@test/suites/c2cc/cleanup.robot`:
- Line 89: After registering cluster-c (the line using "Register Remote Cluster 
cluster-c    ${HOST3_IP}    ${HOST3_SSH_PORT}    ${KUBECONFIG_C}"), add the same
cleanup assertions that currently validate Cluster B's CIDRs/rules are removed
from cluster-a but applied to Cluster C: duplicate the existing Cluster B
cleanup checks and change their targets from cluster-b to cluster-c and from
${KUBECONFIG_B} (if used) to ${KUBECONFIG_C}, ensuring the assertions verify
Cluster C CIDRs/rules are removed from cluster-a.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 6fc450cd-8ba4-4914-afa2-c91e2df14697

📥 Commits

Reviewing files that changed from the base of the PR and between 50d7f93 and 8494088.

📒 Files selected for processing (6)
  • test/resources/c2cc.resource
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh
  • test/suites/c2cc/cleanup.robot
  • test/suites/c2cc/connectivity.robot
  • test/suites/c2cc/infrastructure.robot
  • test/suites/c2cc/sanity.robot

Comment thread test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh Outdated
Comment thread test/suites/c2cc/sanity.robot
Comment thread test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh Outdated
Comment thread test/suites/c2cc/cleanup.robot
@agullon
Copy link
Copy Markdown
Contributor

agullon commented May 20, 2026

this is great work, thanks @vanhalenar
there's another PR adding C2CC tests: #6638 I suggest that PR to be merged before this one, wdyt?

@vanhalenar
Copy link
Copy Markdown
Contributor Author

vanhalenar commented May 21, 2026

there's another PR adding C2CC tests: #6638 I suggest that PR to be merged before this one, wdyt?

okay, I'll wait until that gets merged

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/scenarios-bootc/el10/presubmits/el102-src`@c2cc.sh:
- Around line 44-66: The yaml_content is being built with literal "\n" escapes
and then written with a quoted heredoc (<< 'EOF'), so the file ends up
containing backslash-n sequences; fix by building yaml_content with actual
newlines (e.g., use $'line\nnext\n' when concatenating into yaml_content or
accumulate lines with simple newline characters), then make sure the heredoc
used by run_command_on_vm allows variable expansion (use an unquoted <<EOF) or
alternatively pass the string through printf '%b' so escape sequences are
interpreted before writing to /etc/microshift/config.d/50-c2cc.yaml; reference
yaml_content, run_command_on_vm and the heredoc around 50-c2cc.yaml in your
changes.

In `@test/scenarios-bootc/el9/presubmits/el98-src`@c2cc.sh:
- Around line 44-66: The yaml_content string is built with literal "\n"
sequences but the current run_command_on_vm invocation uses a single-quoted
heredoc ('EOF') so those escapes are not expanded; update the write step to
expand backslash escapes (e.g. pipe printf '%b' "${yaml_content}" or echo -e
"${yaml_content}" into sudo tee /etc/microshift/config.d/50-c2cc.yaml) so
yaml_content becomes real newlines; locate the yaml_content variable and the
run_command_on_vm call to /etc/microshift/config.d/50-c2cc.yaml and replace the
heredoc approach with one that expands escapes (or remove the single quotes
around EOF and ensure expansion will occur).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: b8003f59-889b-4d60-a618-935f9f1b63f9

📥 Commits

Reviewing files that changed from the base of the PR and between 14acc84 and 8bdcb19.

📒 Files selected for processing (2)
  • test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh
  • test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh

Comment thread test/scenarios-bootc/el10/presubmits/el102-src@c2cc.sh Outdated
Comment thread test/scenarios-bootc/el9/presubmits/el98-src@c2cc.sh Outdated
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 21, 2026

@vanhalenar: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-tests-bootc-arm-el9 3e8e730 link true /test e2e-aws-tests-bootc-arm-el9
ci/prow/e2e-aws-tests 3e8e730 link true /test e2e-aws-tests

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 21, 2026
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 21, 2026

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants