Centralised logging via rsyslog

[soreana]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

Systemvm

CLOUDSTACK VERSION

It can be merge to 4.14

CONFIGURATION

OS / ENVIRONMENT

Used Ubuntu 18:04 as a host for management server and KVM Hypervisor.

SUMMARY

Hello Cloudstack community. There is a systemvm remote logging feature proposal. It enabled administrator to access systemvm's log file through management server. I think it would be interesting for other as well. To better explaining my setup I depicted the configuration in below picture.

In this case rather than downloading log file manually from systemvms, they will be available automatically through management server.

STEPS TO REPRODUCE

Below are configuration files for hypervisor, management server and sytemvms.

Hypervisors

Below 4 lines have to uncommented in rsyslog configuration file on each hypervisor in /etc/rsyslog.conf

module(load="imtcp")
input(type="imtcp" port="514")

module(load="imudp")
input(type="imudp" port="514")

Adding below file on each hypervisor to export logs from hypervisor to management server.

# cat /etc/rsyslog.d/00-fwd.conf
if ($fromhost-ip startswith '169.') then {
    *.* @<Management Server IP>:514
    stop
}

Management Server

Just like hypervisors, below 4 lines have to uncommented in rsyslog configuration file on management server in /etc/rsyslog.conf

module(load="imtcp")
input(type="imtcp" port="514")

module(load="imudp")
input(type="imudp" port="514")

Add below file on management server

# cat /etc/rsyslog.d/00-remote.conf
$template remote-incoming-logs,"/var/log/rsyslog/%HOSTNAME%/syslog"
$template remote-incoming-logs-combined,"/var/log/rsyslog/remote"
if ($fromhost-ip startswith '10.' or $fromhost-ip startswith '172.') then ?remote-incoming-logs
if ($fromhost-ip startswith '10.' or $fromhost-ip startswith '172.') then ?remote-incoming-logs-combined
& ~ 

Systemvm

00-fwd.conf file should add on each systemvm as well to export logs from systemvms to hypervisors.

# cat /etc/rsyslog.d/00-fwd.conf
*.*  @169.254.0.1:514

49-cloud.conf file should add on each systemvm. This file describes the logs that eventually sent to a management server

 # cat /etc/rsyslog.d/49-cloud.conf
$InputFileName /var/log/cloud.log
$InputFileTag cloud
$InputFileStateFile stat-file1-cloud
$InputFileSeverity info 
$InputFileFacility local7 
$InputFilePollInterval 1 
$InputFilePersistStateInterval 1 
$InputRunFileMonitor

Need to uncomment below lines if they are commented in /etc/rsyslog.conf

$ModLoad imudp
$UDPServerRun 3914

EXPECTED RESULTS

Management server store systemvm's logs in /var/log/rsyslog/<Systemvm's host name>/syslog

[DaanHoogland]

@soreana are you still planning for this?

[soreana]

@DaanHoogland Honestly, it took a lot of time to apply the changes you required to merging this pr. I discussed it with my colleagues, and we concluded that it isn't our priority now. We focused on other stuff. It is still in our backlog, and we planned to work on it next quarter.

[DaanHoogland]

tnx for your efforts @soreana , i'm marking it as unplanned for now, feel free to come back at any time of your convenience.

[DaanHoogland]

@DaanHoogland Honestly, it took a lot of time to apply the changes you required to merging this pr. I discussed it with my colleagues, and we concluded that it isn't our priority now. We focused on other stuff. It is still in our backlog, and we planned to work on it next quarter.

btw, i hope they didn't sound unreasonable!?

[soreana]

No, they are legitimate requirements :D

[DaanHoogland]

@soreana should/can we mark this for 4.16?

[soreana]

@DaanHoogland When do you planned for 4.16 release?

[DaanHoogland]

this summer somewhere.

[soreana]

I asked our PO. We made a story to work on that. :)

[yadvr]

Removing 4.16 milestone as PR has no milestone; let's move this back if the milestone tagging changes.

[soreana]

@rhtyd good 👍
Sorry for late, we got a lot of issues recently, I can't find a room to work on this pr. Maybe next week 🤞

[yadvr]

I think it may be possible via some log4j config?

[yadvr]

I think it's possible to do this as part of a custom systemvm.iso where you can bake-in your deployment specific option. Any further update on this @soreana or we can close the issue?

[btzq]

Hi, this is something we are looking forward to! Currently monitoring VR is difficult. So we are really looking forward to it!

[vishesh92]

This can be solved by #11654 feature added in 4.22 which allows adding custom userdata to SystemVMs (Console Proxy, Secondary Storage VM, Virtual Router). You can check the docs here.

[daviftorres]

@vishesh92 suggestion is exactly what we are doing. We install Filebeat to export logs to ELK, alongside with RMM tools (for managing and monitoring).

[daviftorres]

Dear @soreana , you idea is sound. I would just be careful on sending more data and causing more overhead on the ACS Management server. Have you considered sending logs to another dedicates hosts (maybe running Graylog), or even a SIEM (like Wazuh)?