Skip to content

feat(system-tests): run firewall_correctness_test on the local backend - #11217

Open
basvandijk wants to merge 3 commits into
bas/local-backend-api-bn-playnetfrom
bas/local-backend-firewall-correctness
Open

feat(system-tests): run firewall_correctness_test on the local backend#11217
basvandijk wants to merge 3 commits into
bas/local-backend-api-bn-playnetfrom
bas/local-backend-firewall-correctness

Conversation

@basvandijk

@basvandijk basvandijk commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

firewall_correctness_test asserts that port 8080 is closed from cloud engine nodes to non-cloud-engine nodes. It was the last test pinned to backend = "farm" by the local backend's firewall setup rather than by a missing feature.

Why it could not run locally

init_ic seeds a Global-scope registry firewall rule so the test driver can reach the nodes once the orchestrator's nftables ruleset is live — the driver's source addresses lie outside the nodes' /64, so nothing whitelists them by default. That rule whitelisted the group's whole ULA range fd00::/8, and ic-prep always adds 8080 to it. The nodes are addressed out of that same range, so the rule opened 8080 between all of them and the assertion could not hold.

The change

The rule now whitelists the driver's own three addresses (management, journald-streaming, file server) as /128s instead of the range they live in. Node↔node traffic is then governed by the registry's node whitelisting rules alone, exactly as on Farm.

Nothing in the IC needs the wider range:

  • non-cloud-engine nodes reach the NNS on :8080 through those same whitelisting rules;
  • cloud engine nodes reach it through an API boundary node's :443get_node_api_urls picks API BN URLs by reward type precisely "since NNS nodes would not accept our connections due to firewall rules";
  • API boundary nodes reach the replicas on :8080 through the whitelisting rules, and XNet :2497 comes from the all-nodes rule.

What can need it is a VM in the group other than the driver. Such a VM shares the nodes' /64, which the GuestOS firewall already accepts on 7070/9090/9091/9100/19100/19522/19531 (plus 9314 on cloud engines and 9324 on API boundary nodes) — so only the remaining whitelisted ports (22, 2497, 4100, 8080, 19523) become unreachable. InternetComputer::with_extra_firewall_whitelist (and the with_group_wide_firewall_whitelist shorthand) widens the whitelist for those cases; the extra prefixes and ports are added to the driver's, so a caller cannot lock the driver out.

Sweeping the local tests that put a VM in the group, exactly one needs it: the rosetta tests run ic-rosetta-api --ic-url http://[node]:8080 on a universal VM. The others are all covered by the nodes' own /64 or talk to an API boundary node on :443; the Vector VM is not started on this backend at all.

The driver's three addresses now come from a single LocalBackend::group_driver_ipv6s, shared by create_group, delete_group and the whitelist, so a fourth one cannot silently fall out of the firewall rule.

Finally, the test reads the current global rules through the new TopologySnapshot::firewall_rules rather than assuming that scope starts out empty, which it does not on this backend.

Verification

backend = "farm" is dropped from firewall_correctness_test, which now passes on the local backend (296s, 5 IC node VMs plus an API boundary node).

Run locally alongside it, all passing:

  • firewall_priority_test_local and firewall_max_connections_test_local — the other two tests over this subsystem;
  • rosetta_test_local — the one test that needs the widened whitelist;
  • cloud_engine_canister_sig_test_local — a cloud engine subnet replicating its registry through an API boundary node rather than through the NNS replica's :8080.

upgrade_downgrade_app_subnet_test_local was not run locally — two attempts died for lack of disk on the machine I verified on, and it is left to CI. It is the interesting case for a released GuestOS: the whitelist stays a Global-scope registry rule, which every GuestOS version honours regardless of the image it boots, and only the rule's ipv6_prefixes change — from one /8 to three /128s, which render into the same ip6 saddr { ... } nftables rule.

On Farm nothing changes: the Global scope is empty there, so the rules the test now reads back are vec![] — exactly what it passed before — and the narrowed whitelist is seeded only on the local backend. firewall_correctness_test and firewall_correctness_test_head_nns were run on Farm at the parent commit and pass; CI re-runs them here.

`firewall_correctness_test` asserts that port 8080 is closed from cloud engine
nodes to non-cloud-engine nodes. `init_ic` seeds a global registry firewall rule
so the test driver can reach the nodes once the orchestrator's nftables ruleset
is live — the driver's source addresses lie outside the nodes' `/64`, so nothing
whitelists them by default. That rule whitelisted the group's whole ULA range
`fd00::/8` and `ic-prep` always adds 8080 to it, so since the nodes are
addressed out of that same range it opened 8080 between all of them.

Whitelist the driver's own three addresses as `/128`s instead of the range they
live in. Node<->node traffic is then governed by the registry's node-whitelisting
rules alone, exactly as on Farm. Nothing in the IC needs the wider range:
non-cloud-engine nodes reach the NNS on `:8080` through those same rules, cloud
engine nodes reach it through an API boundary node's `:443`, and XNet `:2497`
comes from the all-nodes rule.

A VM in the group other than the driver can need it, though. Such a VM shares
the nodes' `/64`, which the GuestOS firewall already accepts on
7070/9090/9091/9100/19100/19522/19531 (plus 9314 on cloud engines and 9324 on
API boundary nodes), so only 22, 2497, 4100, 8080 and 19523 become unreachable.
`InternetComputer::with_extra_firewall_whitelist` widens the whitelist for those
cases, adding to the driver's prefixes and ports rather than replacing them. Of
the local tests that put a VM in the group, only the rosetta ones need it: they
run `ic-rosetta-api --ic-url http://[node]:8080` on a universal VM.

The driver's three addresses now come from a single `group_driver_ipv6s`, shared
by `create_group`, `delete_group` and the whitelist, so a fourth one cannot fall
out of the firewall rule unnoticed. The test itself reads the current global
rules through the new `TopologySnapshot::firewall_rules` rather than assuming
that scope starts out empty, which it does not on this backend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the feat label Aug 19, 2026
@basvandijk
basvandijk requested a lite review from Copilot August 19, 2026 14:35
@basvandijk basvandijk added the CI_ALL_BAZEL_TARGETS Runs all bazel targets label Aug 19, 2026
Empty commit to re-run CI now that the PR carries the
`CI_ALL_BAZEL_TARGETS` label.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Enable firewall_correctness_test to run on the local backend by avoiding an overly-broad local firewall bootstrap whitelist and by reading existing global firewall rules before proposing changes.

Changes:

  • Remove backend = "farm" pinning for firewall_correctness_test and adjust it to hash against existing global firewall rules.
  • Narrow the local backend’s seeded firewall whitelist from group-wide ULA space to the driver’s specific /128 sources, with an opt-in “group-wide” expansion for tests that need it.
  • Update Rosetta local setup to explicitly request the wider whitelist needed for a universal VM to reach replica :8080.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
rs/tests/networking/firewall/firewall_correctness_test.rs Reads current global firewall rules before submitting a proposal so the test can run on local backend.
rs/tests/networking/firewall/BUILD.bazel Drops backend = "farm" restriction for firewall_correctness_test.
rs/tests/financial_integrations/rosetta/rosetta_test_lib/setup.rs Opts Rosetta tests into a group-wide firewall whitelist needed for a non-driver VM.
rs/tests/driver/src/driver/test_env_api.rs Adds TopologySnapshot::firewall_rules helper to retrieve registered firewall rules for hashing/proposals.
rs/tests/driver/src/driver/local_backend.rs Centralizes driver IPv6 sources and exposes /128 prefixes; adds a group ULA constant.
rs/tests/driver/src/driver/ic.rs Introduces API to add extra firewall whitelist prefixes/ports (and a group-wide shorthand).
rs/tests/driver/src/driver/bootstrap.rs Switches local bootstrap whitelist to driver /128s plus optional extras; factors whitelisted ports into a constant.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread rs/tests/networking/firewall/firewall_correctness_test.rs
Comment thread rs/tests/driver/src/driver/bootstrap.rs Outdated
Address review feedback on the local backend's firewall whitelist.

The prefixes and ports render into anonymous nftables sets
(`ip6 saddr { ... }`), and `nft` rejects a set with a repeated element, taking
the whole ruleset down with it. Since `with_extra_firewall_whitelist` adds to
the driver's own entries, a test could introduce a duplicate easily enough — by
calling it twice, or by passing a port that is already whitelisted. Deduplicate
both lists, which also lets `Itertools::join` replace the map/collect/join
dance.

Also record why `firewall_correctness_test` still inserts its rule at position
0 now that the scope is not necessarily empty: both that rule and the seeded
one are `Allow`, so their relative order cannot change the outcome, and 0 is
the only valid position when the scope does start out empty, as on Farm.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

rs/tests/driver/src/driver/bootstrap.rs:85

  • LOCAL_WHITELISTED_PORTS is documented as mirroring the firewall template’s default_rules, but it currently omits port 9314 (present in the template’s default rule ports) and includes 9324 (not in the template’s default rule ports). This makes the comment misleading and can also unintentionally block the driver from reaching node services that are expected to be reachable on 9314 via the local whitelist.
/// The ports the Local backend whitelists for the test driver on every node,
/// mirroring the ports the firewall template's `default_rules` open to Farm's
/// management prefixes. `ic-prep` always adds 8080 on top of these.
const LOCAL_WHITELISTED_PORTS: &[u32] = &[
    22, 2497, 4100, 7070, 9090, 9091, 9100, 9324, 19100, 19523, 19531,

@basvandijk
basvandijk marked this pull request as ready for review August 19, 2026 18:59
@basvandijk
basvandijk requested review from a team as code owners August 19, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants