feat(system-tests): run firewall_correctness_test on the local backend - #11217
feat(system-tests): run firewall_correctness_test on the local backend#11217basvandijk wants to merge 3 commits into
Conversation
`firewall_correctness_test` asserts that port 8080 is closed from cloud engine nodes to non-cloud-engine nodes. `init_ic` seeds a global registry firewall rule so the test driver can reach the nodes once the orchestrator's nftables ruleset is live — the driver's source addresses lie outside the nodes' `/64`, so nothing whitelists them by default. That rule whitelisted the group's whole ULA range `fd00::/8` and `ic-prep` always adds 8080 to it, so since the nodes are addressed out of that same range it opened 8080 between all of them. Whitelist the driver's own three addresses as `/128`s instead of the range they live in. Node<->node traffic is then governed by the registry's node-whitelisting rules alone, exactly as on Farm. Nothing in the IC needs the wider range: non-cloud-engine nodes reach the NNS on `:8080` through those same rules, cloud engine nodes reach it through an API boundary node's `:443`, and XNet `:2497` comes from the all-nodes rule. A VM in the group other than the driver can need it, though. Such a VM shares the nodes' `/64`, which the GuestOS firewall already accepts on 7070/9090/9091/9100/19100/19522/19531 (plus 9314 on cloud engines and 9324 on API boundary nodes), so only 22, 2497, 4100, 8080 and 19523 become unreachable. `InternetComputer::with_extra_firewall_whitelist` widens the whitelist for those cases, adding to the driver's prefixes and ports rather than replacing them. Of the local tests that put a VM in the group, only the rosetta ones need it: they run `ic-rosetta-api --ic-url http://[node]:8080` on a universal VM. The driver's three addresses now come from a single `group_driver_ipv6s`, shared by `create_group`, `delete_group` and the whitelist, so a fourth one cannot fall out of the firewall rule unnoticed. The test itself reads the current global rules through the new `TopologySnapshot::firewall_rules` rather than assuming that scope starts out empty, which it does not on this backend. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Empty commit to re-run CI now that the PR carries the `CI_ALL_BAZEL_TARGETS` label. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Enable firewall_correctness_test to run on the local backend by avoiding an overly-broad local firewall bootstrap whitelist and by reading existing global firewall rules before proposing changes.
Changes:
- Remove
backend = "farm"pinning forfirewall_correctness_testand adjust it to hash against existing global firewall rules. - Narrow the local backend’s seeded firewall whitelist from group-wide ULA space to the driver’s specific
/128sources, with an opt-in “group-wide” expansion for tests that need it. - Update Rosetta local setup to explicitly request the wider whitelist needed for a universal VM to reach replica
:8080.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| rs/tests/networking/firewall/firewall_correctness_test.rs | Reads current global firewall rules before submitting a proposal so the test can run on local backend. |
| rs/tests/networking/firewall/BUILD.bazel | Drops backend = "farm" restriction for firewall_correctness_test. |
| rs/tests/financial_integrations/rosetta/rosetta_test_lib/setup.rs | Opts Rosetta tests into a group-wide firewall whitelist needed for a non-driver VM. |
| rs/tests/driver/src/driver/test_env_api.rs | Adds TopologySnapshot::firewall_rules helper to retrieve registered firewall rules for hashing/proposals. |
| rs/tests/driver/src/driver/local_backend.rs | Centralizes driver IPv6 sources and exposes /128 prefixes; adds a group ULA constant. |
| rs/tests/driver/src/driver/ic.rs | Introduces API to add extra firewall whitelist prefixes/ports (and a group-wide shorthand). |
| rs/tests/driver/src/driver/bootstrap.rs | Switches local bootstrap whitelist to driver /128s plus optional extras; factors whitelisted ports into a constant. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Address review feedback on the local backend's firewall whitelist.
The prefixes and ports render into anonymous nftables sets
(`ip6 saddr { ... }`), and `nft` rejects a set with a repeated element, taking
the whole ruleset down with it. Since `with_extra_firewall_whitelist` adds to
the driver's own entries, a test could introduce a duplicate easily enough — by
calling it twice, or by passing a port that is already whitelisted. Deduplicate
both lists, which also lets `Itertools::join` replace the map/collect/join
dance.
Also record why `firewall_correctness_test` still inserts its rule at position
0 now that the scope is not necessarily empty: both that rule and the seeded
one are `Allow`, so their relative order cannot change the outcome, and 0 is
the only valid position when the scope does start out empty, as on Farm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.
Suppressed comments (1)
rs/tests/driver/src/driver/bootstrap.rs:85
LOCAL_WHITELISTED_PORTSis documented as mirroring the firewall template’sdefault_rules, but it currently omits port 9314 (present in the template’s default rule ports) and includes 9324 (not in the template’s default rule ports). This makes the comment misleading and can also unintentionally block the driver from reaching node services that are expected to be reachable on 9314 via the local whitelist.
/// The ports the Local backend whitelists for the test driver on every node,
/// mirroring the ports the firewall template's `default_rules` open to Farm's
/// management prefixes. `ic-prep` always adds 8080 on top of these.
const LOCAL_WHITELISTED_PORTS: &[u32] = &[
22, 2497, 4100, 7070, 9090, 9091, 9100, 9324, 19100, 19523, 19531,
firewall_correctness_testasserts that port 8080 is closed from cloud engine nodes to non-cloud-engine nodes. It was the last test pinned tobackend = "farm"by the local backend's firewall setup rather than by a missing feature.Why it could not run locally
init_icseeds aGlobal-scope registry firewall rule so the test driver can reach the nodes once the orchestrator's nftables ruleset is live — the driver's source addresses lie outside the nodes'/64, so nothing whitelists them by default. That rule whitelisted the group's whole ULA rangefd00::/8, andic-prepalways adds 8080 to it. The nodes are addressed out of that same range, so the rule opened 8080 between all of them and the assertion could not hold.The change
The rule now whitelists the driver's own three addresses (management, journald-streaming, file server) as
/128s instead of the range they live in. Node↔node traffic is then governed by the registry's node whitelisting rules alone, exactly as on Farm.Nothing in the IC needs the wider range:
:8080through those same whitelisting rules;:443—get_node_api_urlspicks API BN URLs by reward type precisely "since NNS nodes would not accept our connections due to firewall rules";:8080through the whitelisting rules, and XNet:2497comes from the all-nodes rule.What can need it is a VM in the group other than the driver. Such a VM shares the nodes'
/64, which the GuestOS firewall already accepts on 7070/9090/9091/9100/19100/19522/19531 (plus 9314 on cloud engines and 9324 on API boundary nodes) — so only the remaining whitelisted ports (22, 2497, 4100, 8080, 19523) become unreachable.InternetComputer::with_extra_firewall_whitelist(and thewith_group_wide_firewall_whitelistshorthand) widens the whitelist for those cases; the extra prefixes and ports are added to the driver's, so a caller cannot lock the driver out.Sweeping the local tests that put a VM in the group, exactly one needs it: the rosetta tests run
ic-rosetta-api --ic-url http://[node]:8080on a universal VM. The others are all covered by the nodes' own/64or talk to an API boundary node on:443; the Vector VM is not started on this backend at all.The driver's three addresses now come from a single
LocalBackend::group_driver_ipv6s, shared bycreate_group,delete_groupand the whitelist, so a fourth one cannot silently fall out of the firewall rule.Finally, the test reads the current global rules through the new
TopologySnapshot::firewall_rulesrather than assuming that scope starts out empty, which it does not on this backend.Verification
backend = "farm"is dropped fromfirewall_correctness_test, which now passes on the local backend (296s, 5 IC node VMs plus an API boundary node).Run locally alongside it, all passing:
firewall_priority_test_localandfirewall_max_connections_test_local— the other two tests over this subsystem;rosetta_test_local— the one test that needs the widened whitelist;cloud_engine_canister_sig_test_local— a cloud engine subnet replicating its registry through an API boundary node rather than through the NNS replica's:8080.upgrade_downgrade_app_subnet_test_localwas not run locally — two attempts died for lack of disk on the machine I verified on, and it is left to CI. It is the interesting case for a released GuestOS: the whitelist stays aGlobal-scope registry rule, which every GuestOS version honours regardless of the image it boots, and only the rule'sipv6_prefixeschange — from one/8to three/128s, which render into the sameip6 saddr { ... }nftables rule.On Farm nothing changes: the
Globalscope is empty there, so the rules the test now reads back arevec![]— exactly what it passed before — and the narrowed whitelist is seeded only on the local backend.firewall_correctness_testandfirewall_correctness_test_head_nnswere run on Farm at the parent commit and pass; CI re-runs them here.