Migration was refused connection to the destination. Please check libvirt configurations compatibility and firewall

[verendrakalsi]

Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.

ISSUE TYPE

• Other

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

3 KVM hosts on Ubuntu 22.04 all are in same subnet.
Zone :- Datacenter-Linux
Pod :- Linux
Clusters :- Linux

enable automatic DRS in global settings and also on the cluster.

Firewall is disable on cloudstack and kvm hosts.

OS / ENVIRONMENT

SUMMARY

2024-11-22 17:34:27,689 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:ctx-36934979 job-484/job-485 ctx-bb524027) (logid:1ece315b) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.
2024-11-22 17:34:27,690 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:ctx-36934979 job-484/job-485 ctx-bb524027) (logid:1ece315b) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.
2024-11-22 17:34:27,690 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-15:ctx-36934979 job-484/job-485) (logid:1ece315b) Unable to complete AsyncJobVO: {id:485, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAIHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAnNxAH4ABwAAAAAAAAAEcQB-AAlwcQB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345051317061, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Nov 22 17:34:26 UTC 2024, removed: null}, job origin:484
com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.
2024-11-22 17:34:27,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-15:ctx-36934979 job-484/job-485) (logid:1ece315b) Complete async job-485, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ALxNaWdyYXRpb24gd2FzIHJlZnVzZWQgY29ubmVjdGlvbiB0byBkZXN0aW5hdGlvbjogcWVtdSt0Y3A6Ly8xOTIuMTY4LjEwMS4xMDAvc3lzdGVtLiBQbGVhc2UgY2hlY2sgbGlidmlydCBjb25maWd1cmF0aW9uIGNvbXBhdGliaWxpdHkgYW5kIGZpcmV3YWxsIHJ1bGVzIG9uIHRoZSBzb3VyY2UgYW5kIGRlc3RpbmF0aW9uIGhvc3RzLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAFnNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAALA3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAHbWlncmF0ZXBwc3EAfgAMAQAACodxAH4ADnEAfgAPcQB-ABB0ABJvcmNoZXN0cmF0ZU1pZ3JhdGVwcHNxAH4ADAEAABWJcQB-AA5xAH4AD3EAfgAQcQB-ABNwcHNxAH4ADAL____-cHQALWpkay5pbnRlcm5hbC5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbHQAHU5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5qYXZhdAAHaW52b2tlMHQACWphdmEuYmFzZXQABzExLjAuMjVzcQB-AAwCAAAAPnBxAH4AFnEAfgAXdAAGaW52b2tlcQB-ABlxAH4AGnNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AHHEAfgAZcQB-ABpzcQB-AAwCAAACNnB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAccQB-ABlxAH4AGnNxAH4ADAEAAABpcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABXqcQB-AA5xAH4AD3EAfgAQcQB-ACZwcHNxAH4ADAEAAABmcQB-AA50ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnBwc3EAfgAMAQAAAo5xAH4ADnQAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHBwc3EAfgAMAQAAADBxAH4ADnQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVucHBzcQB-AAwBAAAAN3EAfgAOdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxscHBzcQB-AAwBAAAAZnEAfgAOdABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA2dAAPY2FsbFdpdGhDb250ZXh0cHBzcQB-AAwBAAAANHEAfgAOcQB-ADlxAH4ANnQADnJ1bldpdGhDb250ZXh0cHBzcQB-AAwBAAAALXEAfgAOdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ADJxAH4AM3Bwc3EAfgAMAQAAAlpxAH4ADnEAfgAtcQB-AC5xAH4AM3Bwc3EAfgAMAgAAAgNwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ADdxAH4AGXEAfgAac3EAfgAMAgAAAQhwdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAzcQB-ABlxAH4AGnNxAH4ADAIAAARocHQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2VycQB-ABlxAH4AGnNxAH4ADAIAAAJ0cHQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ASHEAfgAzcQB-ABlxAH4AGnNxAH4ADAIAAAM9cHQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAzcQB-ABlxAH4AGnNyAB9qYXZhLnV0aWwuQ29sbGVjdGlvbnMkRW1wdHlMaXN0ergXtDynnt4CAAB4cHgAABCadwgAAAAAAAAAAHg
2024-11-22 17:34:27,726 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-6:ctx-b5d1c146 job-484) (logid:1ece315b) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.
2024-11-22 17:34:27,727 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-b5d1c146 job-484) (logid:1ece315b) Complete async job-484, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts."}

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[btzq]

@verendrakalsi can you explain bit more instead of just pasting the logs.

[verendrakalsi]

not sure why all the details are not showing up, i have updated it.

Anyways.

I have 3 KVM hosts in same subnet and cloudstack mgmt on a different subnet but there is communication between KVM hosts & cloudstack mgmt. There is no firewall in between.

I have enabled DRS at global & cluster level and also they are on same storage as well. However either DRS runs automatic or manual, the VMs doesnt move and i get this error

[slavkap]

@verendrakalsi, are libvirt TCP connections allowed?
The problem is that the connection was refused. This is for RHEL but probably could help in solving the issue unable to connect to server

[verendrakalsi]

Actually i am using ubuntu 22.04 as base OS not RHEL and yes Libvirt TCP connection is allowed

[weizhouapache]

@verendrakalsi
can you check /etc/default/libvirtd if it has the following config ?

LIBVIRTD_ARGS="--listen"

[verendrakalsi]

Here is the screenshot from one of the server and it is same for all of them.

[weizhouapache]

Here is the screenshot from one of the server and it is same for all of them.

can you remove the line LIBVIRTD_ARGS="" and restart libvirtd ?

if everything works, the output should be same as

# ps -ef|grep libvirtd
root         969       1  0 Oct31 ?        00:20:07 /usr/sbin/libvirtd -l

[verendrakalsi]

i have commented the line and i get this output after running the ps command.

[slavkap]

@verendrakalsi, can you check that TCP port for libvirt is open?

netstat -lntp | grep libvirtd

this is in my lab

[root@cs-dev-kvm1-4 packages]# cat /etc/libvirt/libvirtd.conf | grep listen_tcp
#listen_tcp = 1
listen_tcp=0
[root@cs-dev-kvm1-4 packages]# virsh -c qemu+tcp://127.0.0.1/system
setlocale: No such file or directory
error: failed to connect to the hypervisor
error: unable to connect to server at '127.0.0.1:16509': Connection refused

after changing to listen_tcp=1

[root@cs-dev-kvm1-4 packages]# virsh -c qemu+tcp://127.0.0.1/system
setlocale: No such file or directory
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

[weizhouapache]

@verendrakalsi, can you check that TCP port for libvirt is open?

netstat -lntp | grep libvirtd

this is in my lab

[root@cs-dev-kvm1-4 packages]# cat /etc/libvirt/libvirtd.conf | grep listen_tcp
#listen_tcp = 1
listen_tcp=0
[root@cs-dev-kvm1-4 packages]# virsh -c qemu+tcp://127.0.0.1/system
setlocale: No such file or directory
error: failed to connect to the hypervisor
error: unable to connect to server at '127.0.0.1:16509': Connection refused

after changing to listen_tcp=1

[root@cs-dev-kvm1-4 packages]# virsh -c qemu+tcp://127.0.0.1/system
setlocale: No such file or directory
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

for Secure host, it uses tls

listen_tcp=0
listen_tls=1

[verendrakalsi]

As per the documentation https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#kvm-installation-overview
The both should be '0'
listen_tcp=0
listen_tls=0

However i changed it to this and now migration is working.
listen_tcp=1
listen_tls=0

I am going to load more VMs and see if even automatic is working or not. mostly it should.

[weizhouapache]

As per the documentation https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#kvm-installation-overview The both should be '0' listen_tcp=0 listen_tls=0

However i changed it to this and now migration is working. listen_tcp=1 listen_tls=0

I am going to load more VMs and see if even automatic is working or not. mostly it should.

did you add the host via cloudstack UI ?

it looks like your host is unsecure. you can go to host details and click "provision host security keys" which will secure cloudstack-agent and libvirtd with certificates.
please refer to https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#securing-process

[verendrakalsi]

yes i did added it via cloudstacck UI and yes it was unsecure.
I just secured it on all the hosts.
Thank you for this information