ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
Advanced networking, KVM based, VR using the official 4.17 System VM image (Debian GNU/Linux 5.0 (64-bit))
OS / ENVIRONMENT
N/A
SUMMARY
Haproxy log files in virtual router are not released by rsyslogd, causing a full in VR
root@r-5033-VM:~# df -h
[...]
/dev/vda6 4.2G 4.2G 0 100% /
lsof | grep deleted
rsyslogd 342 root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 346 in:imuxso root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 347 in:imklog root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 348 imudp(w0) root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 349 rs:main root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
/etc/init.d/rsyslog restart
df -h
[...]
/dev/vda6 4.2G 1.3G 3.0G 30% /
STEPS TO REPRODUCE
- Deploy a VR
- Add a load-balancing rule to a public IP handled by this VR
- Wait :)
EXPECTED RESULTS
Old log files get released
ACTUAL RESULTS
lsof | grep deleted
rsyslogd 342 root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 346 in:imuxso root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 347 in:imklog root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 348 imudp(w0) root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 349 rs:main root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
Deleted file is held, causing a full on the VR, causing all the deployments inside the network to fail with
WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) (logid:9b4056c5) Failed to create file /var/cache/cloud/monitor_service.json.10cd2d0e-fd3c-
4bc8-9868-7041cdaad464 in VR 169.254.154.26
java.io.IOException: Error during SCP transfer.
at com.trilead.ssh2.SCPClient.put(SCPClient.java:518)
at com.trilead.ssh2.SCPClient.put(SCPClient.java:471)
at com.cloud.utils.ssh.SshHelper.scpTo(SshHelper.java:191)
at com.cloud.utils.ssh.SshHelper.scpTo(SshHelper.java:64)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createFileInVR(LibvirtComputingResource.java:483)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:226)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:244)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:150)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1831)
at com.cloud.agent.Agent.processRequest(Agent.java:660)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Remote scp terminated with error (scp: /var/cache/cloud//monitor_service.json.10cd2d0e-fd3c-4bc8-9868-7041cdaad464: No space left on device).
at com.trilead.ssh2.SCPClient.readResponse(SCPClient.java:51)
at com.trilead.ssh2.SCPClient.sendBytes(SCPClient.java:142)
at com.trilead.ssh2.SCPClient.put(SCPClient.java:514)
ISSUE TYPE
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
Advanced networking, KVM based, VR using the official 4.17 System VM image (Debian GNU/Linux 5.0 (64-bit))
OS / ENVIRONMENT
N/A
SUMMARY
Haproxy log files in virtual router are not released by rsyslogd, causing a full in VR
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS
Deleted file is held, causing a full on the VR, causing all the deployments inside the network to fail with