Health checks failed on VR

[sbrueseke]

I this this kind of alerts from time to time in our UI Alerts:
Health checks failed: 1 failing checks on router r-555-VM / 9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6

In management.log I find:
2025-01-22 17:41:24,521 WARN [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-ee7be78a) (logid:4a205737) Health checks failed: 1 failing checks on router r-555-VM / 9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6. Checking failed health checks to see if router needs recreate

In diagnostic cloud.log I am able to see this:
2025-01-22 17:50:39,837 ERROR Not able to setup source-nat for a regular router yet

This router was started at 2025-01-22 17:39:11
2025-01-22 17:39:11,339 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-81:ctx-48ac6c20 job-20526/job-20527 ctx-8fd308b0) (logid:bde48037) VM instance {"id":555,"instanceName":"r-555-VM","type":"DomainRouter","uuid":"9d2d3c6c-d1a7-4fd0-8143-7a94b7032eb6"} state transited from [Starting] to [Running] with event [OperationSucceeded]. VM's original host: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}, new host: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}, host before state transition: Host {"id":21,"name":"pc-kvm-8","type":"Routing","uuid":"2fbe41ca-73c9-43d4-a16a-1a44be2b1bd2"}

And it looks like the router is working fine. Is this kind of alert expected short after a start or should we investigate this?

Regards,
Swen

[weizhouapache]

@sbrueseke
is it the VR of shared, isolated network or vpc ?

[sbrueseke]

It is an isolated network in an advanced zone.

[weizhouapache]

you can get the health check result by cmk

get routerhealthcheckresults routerid=xxx

[sbrueseke]

Any idea what triggered this alert in the first place?

[weizhouapache]

@sbrueseke
I have no idea for now.

can you share more mgmt/agent logs

[sbrueseke]

We did some deeper investigation and it looks like this has something to do with our backups (volume-snapshots). It looks like one of our nfs-servers (secondary storage) has some cpu stalls in the same timeframe when health checks are happening. We do not have any idea how this is related. What we also are able to see in the metrics of all instances, system vms and virtual routers is, that iops are going up.
Our primary storage is not on nfs and not related. But we do use sds which uses network (different nic), too. Is it possible that the amount of backups is too mich for the tcpstack on the kvm hosts?

This I found in the agent log:
2025-01-31 02:01:19,168 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) (logid:) Seq 19-7793760630141465558: Processing: { Ans: , MgmtId: 165989263229575, via: 19, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.88.32, file: monitor_service.json.772738ed-480e-456f-96d9-f2deb211485b","null - success: Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
","null - success: Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
Error: Nexthop has invalid gateway.
"],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }

[weizhouapache]

if Error: Nexthop has invalid gateway. appears in a BACKUP VR, I think we can ignore it

[sbrueseke]

yes, this is a BACKUP VR, thx for the info.

Any input on relationship between health checks and slow secondary storage?

[sbrueseke]

It looks like we do get this kind of alert everytime a VR is starting. We run 4.19.1.2 at the moment.