SVVM agent is down and unable to upload the ISO

[verendrakalsi]

SSVM logs.

2024-11-11 21:45:34,525 INFO  [cloud.agent.Agent] (main:null) Connecting to host:192.168.100.110
2024-11-11 21:45:34,526 INFO  [utils.nio.NioClient] (main:null) Connecting to 192.168.100.110:8250
2024-11-11 21:45:35,669 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.net.NoRouteToHostException: No route to host
        at java.base/sun.nio.ch.Net.connect0(Native Method)
        at java.base/sun.nio.ch.Net.connect(Net.java:483)
        at java.base/sun.nio.ch.Net.connect(Net.java:472)
        at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:692)
        at com.cloud.utils.nio.NioClient.init(NioClient.java:56)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
        at com.cloud.agent.Agent.start(Agent.java:297)
        at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:454)
        at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431)
        at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415)
        at com.cloud.agent.AgentShell.start(AgentShell.java:511)
        at com.cloud.agent.AgentShell.main(AgentShell.java:541)
2024-11-11 21:45:35,669 INFO  [utils.nio.NioClient] (main:null) NioClient connection closed
2024-11-11 21:45:35,669 INFO  [cloud.agent.Agent] (main:null) Attempted to connect to the server, but received an unexpected exception, trying again...
com.cloud.utils.exception.NioConnectionException: No route to host
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101)
        at com.cloud.agent.Agent.start(Agent.java:297)
        at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:454)
        at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431)
        at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415)
        at com.cloud.agent.AgentShell.start(AgentShell.java:511)
        at com.cloud.agent.AgentShell.main(AgentShell.java:541)
Caused by: java.net.NoRouteToHostException: No route to host
        at java.base/sun.nio.ch.Net.connect0(Native Method)
        at java.base/sun.nio.ch.Net.connect(Net.java:483)
        at java.base/sun.nio.ch.Net.connect(Net.java:472)
        at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:692)
        at com.cloud.utils.nio.NioClient.init(NioClient.java:56)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)

ISSUE TYPE

• Other

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

Agent of SSVM is unknown and unable to also attached any ISO

Also want to add that this environment is hosted on top of the VMware VMs and KVM hosts and cloudstack management are on same subnets and able to go to internet.

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[rajujith]

@verendrakalsi, Does this issue appear in a new Cloudstack installation or a new zone? Has this SSVM worked fine in the past?
Could you try deleting the existing SSVM? a new one should be created instantly.
What about the CPVM? does it have the agent in 'Up' state?
Can you test the connectivity between the SSVM and the management server on TCP/8250?

[verendrakalsi]

Since the beginning the SSVM never worked. Both SSVM, KVM hosts and cloudstack are on same network and even on the internal firewall of the kvm hosts i have opened the TCP port 8250.

[verendrakalsi]

yes, even tried deleting the SSVM and it is creating new SSVM but still same issue

[rajujith]

@verendrakalsi can you run ' telnet 8250' on the SSVM and test the connectivity? If these KVM hosts are nested KVMs on ESXi can you check the VLANs allowed on the portgroup and security configurations to allow promiscuous mode AND forged transmits?
https://www.virtuallyghetto.com/2013/11/why-is-promiscuous-mode-forged.html
https://williamlam.com/2018/04/native-mac-learning-in-vsphere-6-7-removes-the-need-for-promiscuous-mode-for-nested-esxi.html

[verendrakalsi]

Thank you enabling promiscuous & forged transmits on the network port groups on the VMware and rebooting the VMs. Now the agents are up.

Thank you all for all your help.

[weizhouapache]

looks like a network misconfiguration
better to share more details of your env. for example, mgmt/host OS and IP, linux bridge names, public IP range, etc

[weizhouapache]

I can ping the host on which the SSVM or CPVM is hosted but not to 192.168.100.110 or 192.168.100.1

Can you check if ip forwarding is enabled on the kvm host ?

sysctl -a | grep forward

[weizhouapache]

btw, are you using nested environment on vmware ? If yes, have you enabled Promiscuous Mode ?

[verendrakalsi]

Here is the output from one of the node

root@node-a1:/home/admin# sysctl -a | grep forward
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.brens192-207.bc_forwarding = 0
net.ipv4.conf.brens192-207.forwarding = 0
net.ipv4.conf.brens192-207.mc_forwarding = 0
net.ipv4.conf.cloud0.bc_forwarding = 0
net.ipv4.conf.cloud0.forwarding = 0
net.ipv4.conf.cloud0.mc_forwarding = 0
net.ipv4.conf.cloudbr0.bc_forwarding = 0
net.ipv4.conf.cloudbr0.forwarding = 0
net.ipv4.conf.cloudbr0.mc_forwarding = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.ens192.bc_forwarding = 0
net.ipv4.conf.ens192.forwarding = 0
net.ipv4.conf.ens192.mc_forwarding = 0
net.ipv4.conf.ens192/207.bc_forwarding = 0
net.ipv4.conf.ens192/207.forwarding = 0
net.ipv4.conf.ens192/207.mc_forwarding = 0
net.ipv4.conf.lo.bc_forwarding = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.vnet0.bc_forwarding = 0
net.ipv4.conf.vnet0.forwarding = 0
net.ipv4.conf.vnet0.mc_forwarding = 0
net.ipv4.conf.vnet1.bc_forwarding = 0
net.ipv4.conf.vnet1.forwarding = 0
net.ipv4.conf.vnet1.mc_forwarding = 0
net.ipv4.conf.vnet2.bc_forwarding = 0
net.ipv4.conf.vnet2.forwarding = 0
net.ipv4.conf.vnet2.mc_forwarding = 0
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.brens192-207.forwarding = 0
net.ipv6.conf.brens192-207.mc_forwarding = 0
net.ipv6.conf.cloud0.forwarding = 0
net.ipv6.conf.cloud0.mc_forwarding = 0
net.ipv6.conf.cloudbr0.forwarding = 0
net.ipv6.conf.cloudbr0.mc_forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.ens192.forwarding = 0
net.ipv6.conf.ens192.mc_forwarding = 0
net.ipv6.conf.ens192/207.forwarding = 0
net.ipv6.conf.ens192/207.mc_forwarding = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.vnet0.forwarding = 0
net.ipv6.conf.vnet0.mc_forwarding = 0
net.ipv6.conf.vnet1.forwarding = 0
net.ipv6.conf.vnet1.mc_forwarding = 0
net.ipv6.conf.vnet2.forwarding = 0
net.ipv6.conf.vnet2.mc_forwarding = 0

[verendrakalsi]

btw, are you using nested environment on vmware ? If yes, have you enabled Promiscuous Mode ?

I am going to check and get back to you on this

[verendrakalsi]

btw, are you using nested environment on vmware ? If yes, have you enabled Promiscuous Mode ?

No promiscuous mode enabled for the environment as its just a flat network. basically an untagged network from VM perspective

[leo79901]

I think you can take a screenshot like this.

[verendrakalsi]

Here is the screenshot

[weizhouapache]

@verendrakalsi
do you use a single server as both mgmt server and agent ?

[verendrakalsi]

can you please elaborate more on agent? I am confused about that

I have one management server and 6 other KVM hosts.